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PREFACE 



This'Woirklng Note describes the alternatives open to the plpnacrs 
of a federal R&D agency in specifying its organisation and joandgeioBnn 
processes. The alternatives were derived from the practices of 
existing federal R&D agencies — National Institutes of Healthy 
Nati<mal Science Foundation,* Of f fee of Naval Research^ Office of 
Ecbnoodc Ojit^ortunity, Departaent of Agriculture, National Institute 
of ^ntal Health, U.S^ A±x Force, National Aeronautics and Sgace 
Adndnistratlbn, National Cancer lustitiite — as Siey were described ^ 
to us in intepriews. and officinal dpcOinents* However, not all of these 
descriptions have been dhecked by the agencies. Tliis will be done 
•in th€j near future^ . . / _ . 

Thls^ work was cone as part of the effort to plan the National 

■■ M " / ■ ' ■ s, ■ 7 *■ • ■ / , . 

Institute of Education (Ni:^). If authotized by the Congress y the 

.NIE w(puld conduct research and development/ in the field di/education. 

This report is one of a series on; the Institute. The others are: 



National. Institute \ of Education!: Preliminary ; Plan 
for the Proposed Instltfute (RHBi^7-llJW) r 
National Institute gf Educatlbnf Mefch^.^a for 
Ifanaging Fundamental Resfear^ (WN-7i676ivHE^^ 
Natr6nal;ntift^tltute;o Education: KethodiB for 
Hanagltig Prattice^orieated Re^seMch aiid Develoj^f^ * 
meijt (WN-767i-KEW); ^ A v 

National-Institute of Education: Ifethods ^ f O^ \ 
Managing Programmatic Reseairch and Development 
(WN-7678-HEW) ; and ^ 1 * ^ \ 

National Institute of Education: Evaluation Of 
MethD^ds for Mana^g Research and Developinent ; 
(WN-i680-41EW) • . 



Many of the R&D mariagers interviewed during this study expressed, 
the need for ad<lltl«i^at study of the inethods used,in mahagli^g^,^^^^ \ 
military R&D ^in the federa.t gbverhtoent. The lltexatjite bJTi>^his' 



0iibject is sll^t '^n coaparison wltK the llteratuir^ conceraing ttie ' 
maniigeaent of Industrial and mLlltary R&t). This sexles of reports 
seeks to provide a^ basis for reseaitch into inqproved manageiDent > 
practices for nonnilitaiiy fed^.ral R&D, The principal purj^ose of 
these reports, however 9 is to enable the«planners of the National V 
Institute of Educatix)n to benefit f rom ^the experience of other federal 
llfiiD agencies in developing the KIE's R&D nonagenent.prpcedurvis. 
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I. IlirrRODDCIION 

There are two pirlnclpal categories of decision that must be 
.iBa4e in designing a new Institution. The first^ category of decision' 
concerns the institution's static forif — ' the distribution of responsi 
bilitias and authority aioong its suborganizations and staff. These 
df^cisionss,^de£ine the institution's organization. The second category 
of deci8ibn\concems the institution's ^dynaoilc behavior the 
operations and processes throu^ which it executes Its responslbllltii 
and exercises its authorities, lliese decisions determine the institu- 
tion's management. In this report we describe the alt 6mst Ives that 
must be/ considered in deciding iXpoii thie organisation and mandgeaent 
of a new federal agency to conduct and /^Qpport research and devep.op-* 
inent. ' .. ^. ^ - . 

Our s^pedflc concern is with'' the proposed National Institute o£ 
Education (NIE), but the alterhativea we. examine would be germane ' 
to the design, or ^redetflgn, of an R&D agency anywhere in tfie govern^ 
m^nt . Indeed, we have drawn upon the experience of a wide range 'of . 
such agencies, already in existence, to define the spectrum of / v ^ 
possible choices. The organizational >and manag€iri.al altemativeid 
aclopted by the Hational Science. Foundation (NSF) , National Instjitutes 
of Health ' (NIH) , Of flee of Naval Keseaxch (ONE) , Department of Agri**' 
ipulture (U&DA), National Institute of Mental Health (NIMH), Office of 
Economic Opportunity (OEO) , National Aeronautics and Space Altnlnistra^ 

&lon (NASA), U.S. Air Force (USA]^),.and industry have been descilbed [ 

' ' '. ■ . • ■ . if .' : . " 

and evaluated in the four other reports in' thlis. series. This report 

' ' .* . ■■ . ■■ ■ • > 

^ J. Wirt, A. ^ebermah, and R. Levlen, NaHohal JnaHtu^te of '' / 
Eduoatiqn: Methods for Mana^ng Fmdamntal Iteeearohf The Rand/ 
Corpoa^ion, WN-t767$-HEW, November 19^71. J. Wirt ^ A* Lleberman,' 
an(4^]RrTCfevieh» National Inatitute i>f'Bdu6d^ for Managing 

Prcuitice^H^nte^^ The Rand Corpoztaitlo^ 

»ffl-7^77-*HEV» November 19 71* J/ Wirt, A. Lieberman, and R. Levien, 
National Institute of Eduaatioh: Metfrjdei for Managing Progrcvmatic 
R^tlfiaroh ao^d Develd The Rand/ Corporation, :WN-7678-HEW, Novenher*- 

19^1* 3 V Wirt , A. Lieberman, and R. Leyieh, iVatimxaZ JTnstitt^ 
Edification: Evaluation of Methoda for Managing Reaec^^ and' Develdpr • 
The Rand* CorporationV Ww^^^ 



takes the first step-in translating that experience Cor u^e by the 
Natlo|[ial Institute of Education; it collects pd compares the different 
alterfiatlve^ these several institutions hpve selected in making each 
of the critic/il organizational and s^nagerial decisions* ' ^ ' 

ORGANIZATIONAL DECISIONS ' ^ 

Sev^ critical decisions nusty)^ toade^in defining the organiza- 
tion of a fedetal R&D institution. , They are: . - 

li What should be^ the moo or featurea of the organizational form? 
The maior features include linear ox matrix lines of authority^ 
horizontal or vertical scopes of interest ^ ,and centralized 
• ot AecentxsltzeA ^distribution of e 
2. Boo should support of^RSD be ayganCzed? 
w One major fimctloix of federal RiSD agencies is the sz^pport 
of RU> to be conducted .^xtromtii^aZZj/ by n^ 
V tions» universities 9 or ij^duatry* This decision concerns the" 

subofganization bf the agency that pr^Mdes autch support. 
3i Hoto should wri^at of USD be- organised? 

Some »^ but not alX» J^^d^iral RIrD agerrcles also ei^gage in a 
• second major function ;the oonduot of R&D intrcofmrally 
by federal personnel located iti the agency. This decision 
concerns the suborgaa|.zation of the agency that /Conducts 

R&D. - ■ -.v . ^ . ' ' 

4. should adminieirative services be organised? 

Ail federal agencies must arrange ^or the basic fiscal^, 
personnel y infonnation, and budgeting funatibns; federal 
R&D agencies must also arrange for R&D grants and contracts ^ 
management. These decisions concern 4:he ^organization of 
such services'. 

5. What personnel authdriiies are desirable? { 

The 4(^vil Service personnel system was designed tb meet the 
needs of the ^conventional government administrative agency. 
Many gpverht(ient R&D agencies h^ve found that* Its provisioos . 
limit their ability to hire and effectively utilize scientific 



personnel c Many agencies have beedlfg^anted exeoptions or 
]iiDdi£ieati^|j|jB^o£ the Civil Service procedures/ This d^cislott 
concerns the kinds of chang^. in the personnel/ authority 
that ^dght be^ desirable for a new Bgency. % V » 
6^ f/hai^ pex^bnned q^u^^Hea aa^'e de^ • ' * 

This decision concern^ the Jclnds of .skills to be sw^u^t 
' for each of the principal opferational roles in a federal 

R&D agency: extraiaural program of fleer, Intranural researcher, 
administrative officer, 
Bm ahpuld acbieory oounoile be organized?* 
This decision cottcems the mode of organi^zatlon of the/ 
ad^sory councils, •composed of nongoveznment scientiets 
and^^layiaen, that most federel R&D. ^encies employ for - 
guidance and coimmuficatlon. ' # ^ 

Each of these decisions is disctissed in Chapter II of this rfeport^ 

HANAGERIAL DECISIONS 

Six critical decisions .must >e mad^ in defining the mma^ment ; 
"style" of a federal to tostltuti^^ they are: " ' 

1.' yhat ehautd be the niajor featm^e of the managerial style? " 

This decision concerns the dioice of the uliciiation of 
* effort aaorig planning, managing,, aiidjeyaluatrng R&D programs; 
the degree^ of codi^l e^^ 
c the amount of ccmUttoenay parificdpatirm^^^l^ decisionmaking; 
and the variety of management Btyl^a eiz^loyed. * _ • > * 
V 2^ Had ehould budget allocoHom v 

The central management decision is the aliocatlon' oi the ry 
* institution's budget M^ng, iits various activities. This 

: decision concerns the way that allbcation will he determined; 
t.^ HoQ ihould ^^rcgrtxnf^planning' be jponducied?^^ 

, The activities of an R&D institution are ordinarily grouped 
Into pwj^ow. iMs decislo* c<5fncems the methods by whlcj^; 
new programs Vl 11 bcaplaunudt ^ . .'^ ^ v. 



j 



A. Sow BhouZd proj^'^i dev^Zopir^r^ be oork^ted?^ 

Once a program Is platmed^ the {lyoject^ co^'lcpienent it must* 

be selected and isonltored* This dox^i^a concerns the 

' n^thods for such ;8electiW and TOnitoiRng* . * ' f 

5» Hau should. px^grm eifaluat^^i^^^ (Xfndiht'fd?^ - ^ , , - 

Upon completlop of > program. Its re^ts should be evaluated. 

Ibis decision concerns the metliQ^s fof such evaluati:i&h«^ ^ ^ 

6« Hgu^can inQtilM^ioml developmnt he managed? f 

^ " lo addition to direct, xoanageiaent' of t)ieir otim^progratas , 
• ^ ' "■ ■ ■ ■ i ■ >■ ■ , . . 

nany federal^ R&D agencies have\found p.t desirable to 

-* # ' '* ', " ' - ' • , . ' 

decentirallze program manageroent to nbngovemsiental R&D 

• « ' ' ■'" o 

centers , 3j,aboratorles» and imtitutef ; throu^ the .-provision 

of direct institutipnal auppa^tv This deci^on copc'&ma 

^ ):he. nianageiaent of auch 'support • - ^ ' ' 

Each of these decisions is discussed i in -Chapter III of this rcjport* 

CHARACTERISTICS' OF RESEARCH AXP DEVELOPMENT 
* Three major chafactcrlatlcs of an R&D activity ticed^^j>^e 
distiniguished during t^ie-exa«i«jatlon of R&D orgahlzatlgit and manage-^ 
aent alternatives*; * 

•o furpoBei la the RfiD; activity basic^Xpractical, or 

prograraaatic? . • v 

o Sd^^—d-Sithe- R&D activity a syigle project or the 
conjunction of projects into a progr^? 
J o PhoBe: la the RiD, activity In the plannlii^ir , 
^ * * developmentajL » o^. evaluation phase of operation? 

Purpose V ^" < » , 

Many different categorlifl^tlons of RtD activity a^fcordingrto 
• .■• ' ' - • ^ , ^ 
purpose Have been proposed. In thls^ discussion^of ali^emativ^ 

managerial styles » we shall use the categories of basic p pfacttcal,^ 

fiuid progransmatlc^R^* / . 



The central purpose *>f basic research {ox fundamental research) 
is the increase of disciplined khovledge concerning the fundasental 
pbenonena and processes of ' jthe ncitural and nnn-ia^dc world, • The choice 
t!if question to be studied and method of stttdy is priiDaril'y determined 
hy the current st«te of knowledge » the scientist's percept! ot< of 
insportant gaps in underst'anding, and by the available research tools 
Consequently, the best qualified judges of basic research activities 
are ordinarily considered to be the scientists who are i^orking close 
to ond^ arc familiar with the frontier o£ «knowledj^e. « 

The central purpose of pr-acHcat BSD ie^^lmprove^^nt in the 
perfi:^rQiance^ of some ^practical activity organization^ or object. 
Practical R&l) differs from bas^p much ^n substance, as in 

motivation. The choice of question to be studied and^ method of 
study is determined by the current state of practice; by the pj^acti 
tioner's and-scientist 's^'perception of problems, needs, and opportun- 
Itiee; and by" ttje available scientific knowledge and ^ools* Conse- 
quently, practitioners, as well as scientists, may participate in 
judgments and'^choices concerning practical R&D activit^^s. ^ 

* The central purpose of pwgirfic!;waHc ftSD is completion of some 
specific practical task within a finite span of time. Programmatic 
R&D differs from practical R&D not so much in its * subs tance, a^ in 
its finite time Horizon. Practical R&D activities seek ine continuing 
improvement of practice; programmatic R&D a^tiviti^s dre designed to 
achieve a definite goal. The choice ofagoal is ordinarily not a 
scientific question, while decisions concerning its feasibility and 
Lhe means by which it shall be sough^ are. Consequently, both 
practitioner^ and scientists c^y be involved in iQaklng judgments 
and choices concerning prograomatii: R&D activities. 

Scope^ 

While there is no general -definition of the scope of arf R6D 
acti\rlty, common usage t^nds to distinguish between R&D projects and 
R&D programs k ; 

the stballest unit of R&D activity is usual^ly called a project. 

• _ . -^^ 

Projects range In size from che> activities of a single investigator . 



(perhaps oaly part-tlxae) to the ef£o<:t8 o£ teams cosnprlslng a 
^princ>pal investigator and a staff of perhaps 10 or 20^ 

' An^R&D program is the conjunctions of a niuober.of related projects. 
"Hie conjunction may be quixe jclpse, iw that each project depisnds on 
or interacts: With every oihef pit>Ject, or ou^y'he quite loose. ' 
Ordinarily, all the^ projects Will fall within th1^ sasoe scientific 
resi^arch area or be i addressed' to the saioe practical objective. 

^ /• ' y : ' ^ ' V • • ^ 

Phase * . ; ' * 

The process of managing and condiicting R&D f Alls rather naturally 
into .t^Hree distiqibtf and successive* phases i * pl^sming, development, ^ * 
and evaluatibn. kio management paradigms dlffei^ Ih the amount and 
type of effort tl^y tievote'to .^ach of these /phases.^ 

Du^ng the planning phase the basic o^tllhesi of .a; program are 
defln^4 What ate Its dbject{ves?* l(ov much should b^ expended cm ; 
it?., jwh'at pfolocts, should it contain?* HdW; should; they be related ^to* 

each other? " / ' • ^ * ^ 

. * ■ . ^ ... \ ■' ^ ^ . ^ ' ^ • . 

^ - Oujrlng the development, phoae, tffi outllni&s^of th^ program are 
filled in. f Hie performers; of specific projects are ^elected; funds 
to "Suppprtj projects are disbursed; research or disiVelopment is 
performed; and. perf6rm.anc^ Is^ mpi^ ^. ^ ^ 

And during the evaluatim phase ^ the results of progr^ta activities 
are examined to see how veil j:hey liave, a>^ieir^d ;^* program objectives. 

, In some I(2SD management processes^ some of these activities or 
phases may ^estl{^al i>r al^ent entirely (thls ia ^speucifilly true 
of the- planiiiqg and evaluation phdses) »^hile in othexra those same 
phases may receive, considerable eicDphasis. For exaiq^le, most of the 
existing paradigms for ^the'^mana'^ement of basic research devote very * 
little , effort to explicit program planning* and evaluatipn, whiie most 
of the paradigsQs for the .tiwagement. of programmatic RKD lay particular 
stress oh program planning. • / . . 

LIftlTATIONS OF THIS REPORT , , * ^ * * " ' 

In addressing each^of the- o:^ganizational and management dqcisions 
consi^ered in this report^ we have Vonfined our discussion of ^ 
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alternatives alsodt jexcIusiveXy to* those that we Iden^fled In our 
exaBdnation of existing federal it&D agencies. In th|i||vay, ve have 
been able to *t)e explicit about each alternative and |6!draw upon the 
experience of employing it in some practical contextljto estisuaite its 
advantages^and 'disadvantages • At the same tine^ ve jnave been» as a 
consequence » limited in the r^ge of possibilities we ha^^ discussedJ 
Ue recognize that limitation as a deficiency of this report , but 
take seme solace in'xhe observation that the eight f^ederal agencies 
ve have studied eiq>lQy u sqrprisin^i^ biiMi^ structures and 

practices.^ While constrained, the alternatives we describe still 
span a vide range of potisiitlal choices* ' ' * * 
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II. 0RGANI2AI10N IffiCISIONS ; 



WHAT SHOULD BE THE MAJOR FEATURES" OF THE ORGAMIEATIONAL FORM? 

There are four major choices that ]nJ8t l)e made In de£ftilng an 
R&D agency* 8 organizational form. 'These are the diolces between^ 



o 
o 
o 
o 



Linecop and! matrix lines of authority » 
Horizontal and perticat scopes of interest , 
C&ntralized ^d decentralized dlBtrihutloa of effort » 
Integrated and aepora^t^c} location of intramural resea:[rdbi« 



jLines of Authority 

R&D organizations nay have either linear or x&atrix lines of ^ 
authority, r / 

Ifinear . Most conventional organizations hav^e linear lines of 
authority Each staff nenber has one and only qae loinediate jsuper-- 
visor. Each 8 ubofgraization within the* organization has one "and 
onlv one imnediate superorganization. But aincie each organization 
cai^l h^ve several suborganlzations^ the result is the conventional 
pyifaaddal organization chart, Pig# 1/ " 





Fig. 1 Linear Lines of Authority 



This Is the basic !ttlhicture of'jK>8t: RU) sianrage^^ agencies in the 
Federal ^vernsient,|lncludlng NSF, NIHB, and laost of NIH, 
The advantageel hi linear lines of ajuthority are clear ^ 



eath 



staff iDenber knows |f precisely to whom he reports; responsibilities f 
may be clearly define^ arid assigned. Linear lines ox authority 1 
work well when an /organization's responsibilities are reasonably 
I stable and sepdrable into fairly IndependentPsubtaSks, each of whj^ch 

requires a i>redictd>le and reasonably cohist^^ 

For example, disport of basic, research by NSF\is a; reasonably stable ; 
chtfrte^ (despite rapid change within some f i^ds^^^^^^^ 
> fairly Independent stibtas^s (supportNin each' discipline) ^ each^ of 
whlph reqtiires a predictable and i^asod^^^ mixture of 

« skills (program officers conversant with N^he a^ discipMne)« 



The disadvantiagee of lin^^ \ 
conmiunication : and coordination among separate subbrganii&ations 
within ah brganlzatloh of ten pose difficulties (especially when they 
report to different superorgaikizatlons) j siiborganizaClons of ten t6Ad 
to b ecome I identified with and Jverly protective of the external - 
discipline, subject, or problem for whi'ch they have sre^^ 
in bur Interviews, comments su^h as the following * testified to the 
existence of these problems: 



o /^4hen out organization has p repeats with oi^e 

\ phas^ in one di:\^slou and another pha^e in another 

division, we cannot J jet thepcoordiM easily." 
o "Adjacent 'laboratories don't know each other 's*needs." 
o "The training division is only interested in train-* 

ing*mQ(re people; they do not care jhow that relates to 

the/needS 'Of researc 
o "That division is thel captive of its clientele; it 
^ con^rns. itself only with parochial spltitions to the j' 

problems." 



linear. lines of authority tend to work less well when an organization's 
^responsibilities are chan-ging, require the coordination of a number' of 
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different activities^ or demand An uncertain and varying lolx of skills* 
tor example, development of spacecraft by the NASA-Goddard Spaceflight ] 
Center involves, a sequence of changing tasks (each new class of space- • 
craft' requires different technologies) ^ requiring the cootdlnation / . ^ * 
of a nuinber of different activities (ranging fxpm experisiental ptiysics 
throu^ st^ctural and electronic fabrication* to launch operations , 
t^lenetry^ and data reduction) » and es^loying a vide and uncertain 
rmge of skilly (from physics, through various engineering skills to 
blblogyVi^teorolo^^^ and .cpmputex science) , 

Matrix * To overcome some of the lilsadvantages of linear -struc- 
tures, a nu]ii>e;r^of industrial &&D orga^ federal R&D 
agencies have adopted matrix lines of authority* These add a second 
Una of authority 9 usually a flexible and changing one ^ that "buts 
across" the first?* Thua^ some staff meiabers' will have more than 
one immediate supervisor. 'Each suborg^izatlon, however, Gftill*has 
one and: only one immediate supergroup * The result Is a matrix 'of gWi- 
l^zatloh chart/ ilg. 2. itetrlx^rganl^ which In ptactlce do not 
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Fig, 2^ Matrix Llne3 of A-ithori.ty 
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hove quJ.te so regular a structure as suggested by the^ove dlagram» . 
have l>een ^adopted by NASA-*Goddard Spaceflight Center dnd by the 
National Cancer Institute. 

The addaniages of matrix lines of authority are the flexibility / 
that they provide the organization to respond to changing require-* 
nents by est^lishing new suborganizations along the second line of . ' 

V \ _ i ^ ' / 

authorijty while retaining stable relationship^ along the' fir^ ' < 

. • " • . • • ^- ■ ■' . ■ 

*of authority; ind the .enhanced ccTinaiunlcatidin and coordination that they 

•I • . * ■ ■ ' - " .. ' • . • ' ■ • • * . ■ ' . " 

inspire by 1> ringing together ^under the second, line of authority people 
who occupy distinct suborganizations under the first line. 

Matrix organizations have been employed successfully by advanced | 
technology firms in industry, where one linp of authority usually 
follows disciplinary or technical speciality lines (e.g. , there may 
be structures, thermod>mamicc; I 'electronics , and propulsion '<lcpert«- 
.mentcs), while the second llci^ pf authority follows project lines 
(e.g., there may be an orbit'injg telescope project, a Mars spacecraft . ' 
project, am a weather satellite project). Thus', a thermodynamicist'^ ' 
\ might be ^(member of the thermodynamics department and report to 
its head ajj^d also a meisber of the Mars spacecraft: pro^ct and report 
to rits leafier. In the thermodynamics departmtent he Wduld be associated 
primarily with other thermodynaml cists j but in the Mars spacecraft 
project he would associate with the mix of specialists, from other 
depiartments needed to carry on the project « And while the department 
association is stable axud continuing (like the disciplines), (he 
project association is. usually transient and finite. Matrix organ- 
' izations are able, therefore, to respgnd to the demands of a changi4|||f 
environment while also maintaining a stable basic structure for * 
staf fing /and developing competence. ^ ^ - 

the dieadvantagea of matrix lines 'of authox;lty arise from the'''^^ X 
possible conflicts of .atithority arid responsibilfty between t<he two ^ . > 

' dimensions, Vthe possib'le mijsmatch in skills reqijS^red: by tl^ie^ ^ - 

along the two dimensions, and the tendency for ah organization to . > 
revert to a linear line pi^ authority. The firs fo difficulty takes 
its most poignant form fot the staff fheinber who fli^ds his departmental 
> supervisor in conflict id-th his project supervisor coricetning the • C 



quality of his wo rk, his appropriate rey^ or his tenure, but 
It: nay. also aflfect staff asslgnitDe^ty budget idlocatlony and project 

evaluation dedslbt^s. The second difficulty can t)ccur If the range 

' ' * • '1 * " * • ' \' ^ • ' ' '■ ' ^' 

of activities :<er^. y projects) \indettaken along the second dlisehslpn 

fluctuates so broadly that it Imposes widely varying staff reqiilre-* , 



nents on th^ units of . the first 41iDehBion (e.g.., departwents) The^^ 
third difficulty Can occur If « for example, the second dlinenslon 
becomes stablciy with staff assignments and subunlts unchanged oyer . 
long periods* Then, the matrix structure becomes two fixed i, coiqietlng 
bureaucrecie^, one of Wich will probably p . 

Undoilltrtedly, matrlk ^organizational • forms reqxtlre greater skill 
frcnn their managers, b^ecail^e the coordinating ahd balancing o£ over- 
lapping 'authorities and respohslbllities must occur continuously. 
However, .HASA's Goddard Spacej^llgbt Center has ^ievelpped several 
mechanisms; to alleviate those difficultly:^. Which make the manage- 
ment' problems less ^severe • For exasple, at Goddard a ataff member 's 
adminiktrative and physical locations change or not depending u|»oti 
his roXe-ln^^ project, rAdinlnlst rat ively^ he may be ei^thco: ^^solld- 
y^lne" or "d6tted-Une^^f^^t^^^^ project • \ lf he is. solld-Une, th^n he . 
is physically and administratively colocated, wlj^^^the project, ^d 
his career advancement depend^ on the p&ject; director; if. he 1| -i^V^' 
dotted-lliiev then he will physically col^cate with the pro j diet > buit 
administratively remdln with his departmei^t, and his career aj^yance- 
ment depends on his department head.' For ^1 spacecraft projects, 
the system-level $taff are solid-line Co thW project, as are son^ 
subsystem-level staff ; meat component-level ^ taf f are dotted-line 
• to .the project* : .For tJi^ e^en subsysteri^level 

people are dotted-llne. . * • - * * 

Goddard also avMds the second difficulty or ni^trix management 
(mismatch oi?' staif skills and needs) because its stai^f is large and 
quite diverse /r iSbddard^s managers emphasize/that "staff diversity 
is the key to miitrix management; you must/have the competencies when 
you need them/* In addition,^ the choice^ among prospective new, 
..projects is affected by the availability of appropriate staff • The 
National Gancer Institute Has adopted a somewjhat different approach; • 



1 



-IS--* 



it avoids mismatches between, staff availability and needs by drawing 
upon outside scientists to fill certain posltlims in the program 
organization* ^ \ 

The third difficulty in matrix 'organization (reversion to a 
linear line of authority) can be avoided by assuring that one line is 
constantly changing in its suboifganizations (e.g*, tasks or px^iects) 
80 that it does not ngidify into iiimply. another fixed way of organ- 
^izin& the same popple; the flux and constant reorganization of one 
line is what gives a matrix organization its ability to* coor4inate 
effort on a dianging series of projects or problems. 
• ' » .. • . • 

Scopes of Interest . 

The cpre of ah R&D organization's l^cope of iuter^t is its 
collection of ^R&D projects and prograiins* -We shall iconsider an R&D ^ 
organization to have sit^JioHzonial scope of -interests if it builds 
tin that core collection solely through the addition of R&D projects 
and programs pn new topics (as has NSF) . An R&D organizati^ has a 
veptioql scppe of interest if it builds oq that core t^foijiglti the 
addition of r . 



o 
o 



dlssemination^activities (as hoi the Department of 
Agriculture) ' ' " \ 

service and training activities (ad has the NIUH) 
policy planning activities (as^has OEp) 



NSF 





^ 




R&D - 


H&D i 


R&D 






{ Disseiain-l 


• ■ 




1^ atipn j 


a.' Horizontal 








V • 



USDA. 



NIMH 




Jb. Vertical 



OEO . 

r"""'"~~^-~"i 

I Policy , I 
{ piling j 




Fi'g. 3 Horizontal and Vertical Scopes of Intere's-t 
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. Horizontal. * Those vl^o argue for a horizontal^ scope of Interest 



feel that: 



o if 8^n>iae or polid^/^ing iiinctiona are add^d, / . ^ 

the organisation finds/ operational priorities ^ 

Overbalancing research priorities, demands for 

^ Inmedllate results driving out basic and long-term 

research I and demands to s^^port decisions already 

taken supplanting the neutral stuay of fu|;ure 

alternatives. / - 

6^^^ training or diaeemination functions are added, ^ 

^ ^i^d^rganlzatlbn finds Its energies diverted by the 

hea^^ efforts essential for effective training and 

■ ' • . * • 

dissemination. 

' • * ' "V ' 
o only if it restricts its concerns to R&D activities 

alone, can tne^ organization satlsfy' the ndeds of 
sdencetand erratare proper ^t tent ion to (basic, long- 
term, and controversial |t&D. 

*^ 

Vertical . Those who argue in favor of some form of vertical 
scope of interest feel that: s V 

• ' • - ■ • * ' . " . — 

. ■ . • ■ ■ *■, 

o linking R&D with dissemination » service, ,br policy* ' 

• * ■ 

making is "very effective In generating political 
(and flnanc;Lal) support^^or, the organization's - 
programs (including: R&D) • ' * 

o such linkage "forces the nasty questions what c ^ 

do the research outcomes mean for service? ~ ' ' 
^ ' to be 'asked by the organization's own staff" and, i 
therefore, to be taken seriously* Without it, \ 
research tends to drift away from practical needs.* 
o linkage between R&D and dissemination is crucial 
to success o6 both activities; because it provides 
research with constant awareness of problems 1. 
needing' solution and provides dissemination with 
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V ; W'Winually xefljjwhed pool of 



o Ucdcage het^if^^ and pollcyzBaklng (espetLally 
^ though pai^i^^^atlon of K&D "managers In policy 
Studies) bibll^iKU) findings Into policy influence , 



and assures 



R&D t/ill address pollcy<-relevanjt 



topics. v/?:r J . , 

Three specific of vertical RjU) organizations axe: 



NaHonat _ 




iitute of. Mental Health ^ 
NIHU coo^ijpps Tesponsibilities for research, training* 
an^ sen^^ (through support of Conaunity Mental 
Haalth C^jfi^^ Its internal ^struc^ui^ however » 

is a co^l^ation of horlstqntal and vertical* sub<- 
structuie^l^ Fig. 4<r 



Hesear^ 



Training 



\ 



^ 

Service 



Fig. A National Institute of Mehtal Health 




There are separate internal divisions for research » 
se'rvlce, and ti;aining» but there is also a division 
that contains centers concerned with specific ^ * 
problen areaS; (e.g.:,: crime and delinquency ^ minority . 
studies* me tropolit ah stiidles)> some of which combine ; 
research^ sergice» and training functions related 
to their^ probli^m ar^as/' ' \ 



Depctrtmnt of Agriculture 

The success of the Agi^culturaJf. 1^ system has been 
cryptically dependent cm the close linkage ainpng 
three different types oif specialist: < ' 
^ o oomty agents (serving the f an&er by 
helping him identify his "problems and 
x&aking available to him th^ findings 
of research) , 

extenaian apedaliatB (serving the 
county agent as a source of specialized^ . 
iq)-^to^ate knowledge about researdi on 
a particular cozonodity dr problem^ and 
^ transmitting n|^ prolUems to the 

researchers) 9 
o reeearchera (pursuing new knowledge and 
the solution of practical problems)* 

The three typed have been essential because: the 
•county agents alone quickly ua^ up their store of 
research knowledge and lose the f armer^s qonfi*-. 
dence} the researcher alone cannot communicate 
directly with the farmer; but the county agent also 
has diffictdty communicating 'directly with the 
researcher; thus, the extension specialist serves 
as a cnidal middlepm. These three functions are 
the dlre^ct responsibility of the Department of 
Agriculture in varying degrees* Thfsre is con- 
sixlerabl^ overlap with both state and local agencies 

Offioe of Eoonomic Opportunity 
V The Office of Planning, Kesearcht and Evaluatlo^ 
. (PR&E) of OEO has a duql.roler it serves both ' 

as a pi^Qping staff to the OEO director and as 
' the managisr of a researdi and evaluation «progi^ 

that has a strong policy orientation .and whose 

findings exert a direct influence on policy. ' 
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Both the National Sdience Foundation and the National Inatitutee 
of Health dre^ vlth slight exceptions » exasples of R&D organizations 
with horizontal scopes of voxk; 

platrlbutl#n of Effort . . l ' 

A ^^tralized R&D or^nlzatibn is one whose R&D activities are 

carrled'^Jlit at a Single iKtealc imder the direct authority of a single 

*■<.■■ 

agency executive.^ An R&D organization may he decentralized in 
several senses; ^ 

^ o > regional location ^ portions of the organization are 

at different geographic locales* but they report to * 
the. sane agency, executive « ^(For ex^siple, regional 
^ offices of JIIMH.) 

o tegipnal o^t^^ration — portions of the organization 
are at different geographic locales czn/2 each has 
Independent operating authority, through the provision^ 
for exa&q>lo* of fonmila^graht |unds (the fifty-three ■ / ^ 
State Agricultural Experiment Stations are fonmxlar 
funded and receive state matching grants)* 

o dual operation parl^^ the organization is 

centralized » part decentralized* 

/ . -a . - 

The AS^^l^uf^ R&D system is dual it comprises both ^ 
centralized and decentralized R&D activities. This is frequently 
cited as a principal strength of thd agricultural R&D system. \ A 
' V ' ^ 
o T^^ AgHauttuTal EeeeaMh Seraioe (ARS) provide^ 
centralized management of USDA*s intramural R&D* 

about 50Z of the total R&D budget. Hils coorditiated » 

\ • 

concentrated Jl&D is conducted at facilities distributed 
49iround the country » many colocated with universities. 

o The Cooperative State Researdh Service (CSRS) provides 

formula grant support (on a state odtching fund basis) ' . 
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to State Agricultural Eacpertiaent Stations (SA£$) • \ 
each state and -territory. 

The ARS laboratories are assigned problems of n^itional scale. , They 
can form a large teaa easier than CSSS can^ can cover all phases* of 
R&D9 and can bring problems to the pilot plant stage. They oan act 
^^s^ill gaps In the prograns^^rrled'^out with little coordination ^ 
by the^ s't^ations. The CSHS state e^rlioent stations ^re fre^ to 
expend thel& formula grant son^ as they ylsh on proSllig^^f r'coQcem 
In their state. They . act to k^ep the systea fully aware of ^at the 
probleos are. However ^ they have a tendency to dl^s^ate their ' 
efforts by putting a little in eadi area and to ditpllcate the efforts 
of other stations. The result Is often soultlple problem solutlcms. 

The df'centrallzatlon of USM^s B&D program provider several 
d^atlnct benefits; ^ ' ' ' - 

o the states and counties pay part of tjhe cost of *8upjport» . 

and are genf^rally strong advocates of agtlcMltural j(&D« 
o' Congressional support* Is strengthened because of the 
location of R&D facilities In many congressional 
J^BtxictB dsXd l^ benefits to farmer constituents. 

the decentralized facilities are* aware- of and respon"- 

» - •> ■ * , ■ 

elve to local need^. 

The tension between national and local inte;rest in USDA's 8&D program 
alap provides several benefits: " ' 

■ " m ^ - ' 

'< ; • ■ ■ ^ ■ *' ^ 

^ o ; it promotes and broadens compet'itloc^ In serving the 
farmer* * . ; * ' 

o It broadens the R&D perspective tn( include bod\ local 
> and national problems « * 

o it provides a isiechanism for communication between 
. /.^national and Ipcal RU) efforts./ 



NIMH has a regional crirganixatlon* but the regions prlaarlly have 
*■ ~ * * . ^ 

a service orientation; their Involvenent in research is slight and 

decreasing. / \ 

ilost federal R&D agcncie^^ are centralizltd. NSF» NIH an^ OEO^s 

PR&E man^e their research and conduct it (when they have intramural 

activities) in a central locale (with the slight exception. of NIH^s^ 

environmental » institute in North Carolina).*. They .benefit frcHn the 

advantage in coordination and codjitfration that geographic continuity 

provides. In the case of I^SF and NIH» constituency' Sjupport is'^uilt 

through tlje hig^ily diatribut.ed expenditure of ei(t|:amur#l grant and 

contract funds . . ^ 

Location of Intraimiral Itesearch \t 

Host Federal R&D organizations having botK intlramural R&D 
performance and extramural management responsibilities, s^pomte 
them at the highest level in the organizatiOii^ Intramural research 
at most NIU Institutes and at NIKH is geographically and organiza^ • 
tlonally distinct from extramur4l re^ardi support. The opposite' 
extreme » complete tnt^^jrat^m of intramural research and inanageiaent 
of extramural R&D igTo that they are done either by 'the same' individual 
or by individuals adjacent both geographically and organizationally » 
is relatively rare. However » NASA«*Goddard/ the National Cancer" 
Institute (NCI) and 0£0*s PR&E 'have accbs^lished 4 1 intone form 
or another. - TUelr or|ffi)iz jtlonal structures are shoon in Pig. 5. 

Separate. In the VIR Inetitutea there a separation immediately 
belov'the Institute Director between intramural axvi extramural 
researeht a different Assistant Director Having redpQoaibllit;/ for 
erach. * This isbparation mirrors a separation slc the overall NIH 
le^el. . ^e« separation is sharpened by th^ f act that the intramural 
research faei^lities of all the Institutes are physically adjacent on 
the main » caucus, while the extramural offices are geographically 
'6eparat:e. ' Moreover » the intramural reiseardi directors x>£ all the 
InstiJtutes Jteet together Regularly u^der the aegis of the NIU*s 

overall Intramural director. The"^ communication between Intramural ( 

- * ^ . - ' " " . 

and extramural personnel within an institute is* less regular*; The. 



NIMH has Inherited a sijsdlar pattern txom its another organization, 
including the geographic separation of Intranrural and extramural 
facilities^ 



Director 



Assistant Director 
Extramural 



1 



Assistant: Director 
intramural * 



a. The NIU/NlMH Model 
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b. OEO Mod€^l 



Fig» 5 location of Intramiral Research 



The adomtagea of .intramural separation are: 



o different personalities and styles of work are 

required for intramural research in* the biomedical 
sciences, and the management of extramural research; 
.they cannot be done by the same person and different 
.people ^j^g th%m do r^ot work closely together, 
even if placed together. 
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o intramural biomedical researchera are scientific 
"competitors" of the extramural researchers seeking 
support and musty therefore, be kept from seeing 

proposalis. ^ 

' ■ .• . ■ 

J Integrated , In the OEO 's Office of Planning; 'R^aeardh and Eval-- 
naHvn (PRiE) , a fair degree of integration between a jfoodest intra-» 
mural HfiD^^aptivity and 'extramural R&D programs ^is achieved- through 
the close Interactions among a small staff dlvldea along fuuctional 
lines: - .one group concerned with experimentation^ one with evaluation, 
and one wltK policy^ reseerch. Each of the first two groups does 
considerable extramural funding ^ research, but project monitors ofte^ 
participate directly in the project or carry on their own research, 
-Intramural researchers both have close contact with extramural project < 
monitors and may contract for work that their research suggests is 
important, but that cannot be done intramurally. 

The adpantagee of this form of <*intramural/extramural integration 
are: . ' - . ' . ' ; .- , ' ' ^ ' 

o «the PR&E staff may focus on specific policy questions 
and seek answers by whatever meaits intramural ot^ 
extramural — is appropriate to the question. 

a the existence and availability of intramural S&P 
competence improves the quality of the planning, 
selection, and monitoring processes for extramural . 
RiD . M ' ^ , 

o intraiSural researchers provide a' direct channel for 
th£ findings of extramural R&0 into policy v^ecisions. 

. In NASA's Goddiard Spaceflight Center and at the Nation^ Cancer 

Institute i the matrix organizational form is used to integrate intra- 

mural and extramural R&D. Intramural researchers report along one, 

■ ■■ • o . • 

stable line of authority according to their discipline. However, 

they laay be assigned to a project team managing extramural R&D along 

a seconcl, flexible line of authority. In ^he first role, they dp 

intramural research; in the second role, they manage extramural 
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, P&D contracts. At NCI 90% of the Intramural staff Is also managing - 
extramural research as part of a project team. At NASA-Goddard there 
Is a conscious management policy of moving people from Intramural 
research to extramural management respdnslbllltles. 

Vxe, advantages of this form of integration- are^ 

'...'♦••*'■ . ' ■ ■ 

' o the existence of in£ramural expertise- enhances the , ; _ 

quality of extramural pro j'ict .management. : - * ' 
o intramural research competence is extendeii by the 

1' ability to contract! for extramural work and by • _ . . 

'contact with it. 
o attractive career^paths are opened to staff ifno may 

grow through Intramural research to a position ^pf • o ^ ^ 
researdt management responsibility. .... . . - 

^ • ■. ^ ^ . 

The NIH InstituteB also support "collaborative" research projectn 
which are ordinarily carried out under contract; as opposed to the 
extramural research conducted under grants. Collaborative research 
originated as a way in which intramural researchers copld contract 
for work that related to or extended their own work, but that could 

not be done dntramurally. The testing of thousands of chemicals in.. 

• * , ..." ' / 

the cancet chemotherapy program is an exaTDpl4. > Thus* collaborative 
research is a' way in which intramural and extramural activities are 
; integrated at NIH. However, it is important to note that it Involvesr 
the contract mechanism and the performance of tasks specified by the^ 
intramural researchers. 

J) ■ ■ , • 

We conclude that the mores of the scientific community make It ^. 

difficult to integrate intramural and extramural' research of the 

kind usually funded by grants in response to unsolicited piroposals. 

Attdiq>ts to achieve such integration must, at least, include 

mechanisms to insure the preservation of scientific priority and 

account for the different personalities and styles of work of 

• active reseaifchers and resea:rch managers. ^ * 

' However, there are several apparently successful models of the . 

integration of intramural and extramural R&D of the kind usually 



carried out under contiract In response to agency- defined statements 
of work. That Is, developmental, pro ject-o/lent6d, or policy- . 
relevant A&D are more likely to be hospitable areas f or Intramural- 
extramural integration. 



HOW SH OULD SUPPOR T OF R&D BE ORGANIZED? • 
■ . . We shall focus on structures for the management of extramural 
research (including integrated intramural research) • \ Structures 
for thie management o€ separate intramural r^ejs^arch will be discussed i 
in th^ next section. There are two .principal deoiaiona 'that^jj^at be 
made *in specifying the organization's internal subs truttute. These are: 

. ■ ;. " ^ - ' ■ ■ ' ■ , * . \ 

-••.-*■.....,. ^' • ^ o . . . ■ . . ., ^ . . . • . . 

. o ; ythatyiaxonomy will be ea^loyed in :defining 

siiborganizatlon responsibilities'? A V ? ;^ 
; o \fhat rolea and funoin^cma will each sub.organ- 

ization have? . ^ < ' 

Taxonomy;* ' * • •■'^ • ' * . ' .' " ' . ' S.^;r •.' 

Most, organizations are subdivided into suborgani-zations, each 
of which (in the ideal) has sole responsibility ta. carry out cl^ar - 
roles and functions in a sharply defined subarear. The assigmbent 
of r&sponsibility is made, according to a^ooidne^ that 3eeks to ^ . 
di^de the organization's area of responsibility into a set of date- 
gories that .are mutually exclusive and esdiaustlve. "Some organiza- 
tions, however, ihtentlonall]^ seek a degree; of overlap of responsibility 
aniong their subdivisions* One recently formed Institute ha^ 
consciously arranged a significant amount.; of overlap among its 
subdivisions in order to avoid the carving out of .separate territories . 
and. to permit: a respdnsibirlity to be easily shifted to anc^he^r 
subdivision if the performan&e of the fir^t is bad. The intent is 
to have 2'-3 functional areas in' which iany given program could fit, 
and if it does not work In one, to try another. The price paid 
for such an arrangement is a great deal of internal conflict; a 
benefit appears to be that staff flexibility and initiative are 



pifomoted. The results of this recent experiment are still not 
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•• V ' " . ■ ■ . ■ • : 

detenolned. We shall continue our discussion as though the ideal of 
ia mutually exhaustive and exclusive taxonosy were being sought « 

Several d^Uferent 8iJ)divi8im tcpconomies h^ve been eaiployed by 
other federal R&D agencies t br are suggested 'by their experience. 

They ;^ are subdi vision by: 

■ ' . • ■ . » . • , 

o Diadplinea (e.g., psychology , sociology , economics) 
Prvblem (e.g. > reading disabilities » inadequate^. 
, financial tftqpport ' 

o RAD Functions (e;.g. » intramural research,^ extramural s . 

research, training/ dissemination) . 
o Sub jecit Area Fun&time (e^g.^ instruction,, assessment, 
r . admlnlsJ:ration) ' ' ^ 

o ' Vlierit OrganizaHona ie»g* ^ element ai^ /secondary, . 
hlgher> preschool, adult) • 

Discipline , * Subdlvisipn by discipline occurs at the NSF. itiEk 
maior advantagea He in the management and conduct of basic research, 
which is ordinarily carried out along disciplinary/ linea l 
. . tie^ and other extramural research organizations. Because of its 

isomorphism with the disciplinary subdivision of universities, thia ' 
siibdivision taxonomy also Is advantageous in recruiting high'^quality 
scientific Staff. 
. The v^jor disadoani^es ot^^6^ siib division are the 

' difficulty of forming the interdisciplinary teams needed for many . ^ 
applied abd problem*-oriented research topics, and the tendency of . 
staff members to seek and be Judged solely by the saoctlons of the 
' . external discipline coinnunity to the detriment of other organisational. 
' ^goals.- ■ •■ . . -V ' ■ , . ■ ■ 

jProblem. Subdivision by probleinii occurs among the NIH insibi-^ 
tiitesi at NASA*-Goddard (if specific^ spacecraft projects are considered 
to be problem-solving activities) / within the Agricultural Research 
. Service, and within the Division of Special Mental Health Programs 
'X' -..at/NIMH;'. ■■• • ' • , .oV \, ■ ■'•'.^V .. 
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-The qdvcqjtag^s of problemrorlentatlbh are that it facilitates: 
the conduct of R&D requlrlnig ioixes of /disciplines » encourages In ter^ 
action among the di8cipliiii^.> has' strpng "pb^^ 

funding :authori ties aiM ccfnstituency groups ^ and smoothes the path 
of K&D products into pT^actice. * < 

The dC&advantageB of problem-orientation kre i:^^ tendency to 
favor shortr term over longer*-te^ to ^ 

pc/li tidal presslxrc^^ ^^b^ wl tM^ii'^^ and outride ai^ the 

difficulty bi staffing and managing problemroriented S&D organizationi^ 
(especla^^ j^. the be^^ " V 

Sxpefien'ca^.withvattG^ subs true-- 

tureSv^tt NIH Institutes sugge^^^^ a > 

. -r • • - A ■ 5* . ■■ - . : ' ^ ■■ - 

- o Income casesv the reorganization will be solely ^ 

cosmeiicf>;^^^ ° 
" ' _ program directors will still bia; chosen for their 
•r :. disciplinary con5)etence.^^ ' ^ v >. 

b '^Despite iiiitial''featG^^^^^^^ 

'^l-'^^. IjJihibited./'' ^.^ i;', - .^-^^ ' t^'^ -f ' ' f - " 

b. Ekttamuxal.^taf|;^^^ 
. a The bas^ic 1^ tlie NI^ 

jV . i)irept6i|?a offide wi^ b^incon3^teAt: vitK^ptoblen^ / 
& ^Wfg^^ joint^ ini^rapiral^fextra™ r^i^ea:^ 

o!tiIH act and 
problems ^mse'iuse c^iplexelil^'pl^^^^ a role* ?^:11ie sub^^ Intp^ 
institutes is^acdprid^^ problem err disease t£|xqfnoiny A 

(e.g.«. Cancer#He£u^i^ and Ketahplic Diseases » 

etc.) 9 while the majjor'^^^^ the . 

Study sections organized, by th€t*:|ja:^icn%f .1^ 

defined according to al^^kcip^itijli^. taxbhony ;/ T^^^ duaUtyvhas v 
proven to be a highly effecH^e- the InsMtute*s 

disease orientation geherajt^s^^cohsldei^^ 

i&d public support, while th^ disciplinary study sections have built 



a strong- constitttency among the widely-distributed biomedical research 
.commonity. Th% NIH Institutes differ considerably in the extent to 
which 'the nominal* dlsease^rientatlpn is 'more* than a poet hoc xlassi*-* 
fication b£ clusters of disciplinary research activities. 

RtJa Functions . Siibdivision according to RfiO functions occurs 
at the Nllfii and NIH, where Intramural research, extramural reseatdiV 
service, and trathl^ are managed through separate JQlyis ions; at 
the Department of Agriculture, wheris research, extension services*, 
ond operating" programs are the responsibility of separate divisions; 
at OEO*s PR&E, where experimentation, evaluation, and policy research 

are separate ji^roupd; and. at NSF, where applied research,, education, 

' . ■ - * ' • .* ■ 

and basic research are distinct diyislons. 

' ('* ' ' * ' . 

The acb>antage of such a subdivision is that it reflects the : 

different management styles, personnel skills, axid client group^ 
associated with each function and pemiits each subunit to brgazdze 
so as best to carry out its job* . . v 

^ The diaad^miage is that it can make very difficult the coordina** 
tlbn of several instruments (researdi and trainingV basic and applied 
research^ research and service) to solve paiTtlcular, problems • * 
The specl^lbts within each subdivision will tend to see only those ' 
aspects of a problem that concern them and to see them only within 
the context of their own plans. • 

^ Ag^lcultfure has overcome some of th^ problems o^ separation of 
functions at the federal level. through the/ extension 
specialists and rasearchers at the state level •.CQEO^^ PR^ avoids . 
most of the problems through' its small size and care'fuf overall 
manc^gement. 

Subjeql: Area Functions * Subdivision acdording to subject area 
functions occurs to some extent at the Agricultural Kesearch Service, 
whejre there are divisions concerned with regulatipn and' control; 
nutrition, consumer and industrial use; marketing research; and ' 
faz^ research* 

* : The cdvcottage of thia subdivision is its correspondence tp the 
Inajpr * areas of continuing concern to u^er groups^ and the incentive, 
therefore, to develop continuing programs of R&D sersiflng t^^ ~^ 



^The disadvantage Is ;t:he prospect that problezss that Involve two 
or loore functions will npt be handled well and that basic research 
serving niore than one function will be ignored. 

Client Organizations . Subdivision according to client organ-* 
izatlons does not occur in the organisations studied , although it is 
natural in both education (elementary/secondaxyy ' higher, preschool, 
adult and continuing, vocational) and crii&inal justice (police, 
courts, corrections). 

The principal ad:o\mtages would seem to .be the close correlation 
that is thereby likely to obtain betwedi the R&D program undertaken 
and the interest and problems of tlie eventual user organizations and 
the probable strong political, support that that closeness 'will produce. 

The major disadoanta^ea' Bxk the possibility of co-*optatlon of 
the R&D program by ^sxlsting organizations, inhibiting the development * 
of R&D activities that too. severely challenge the a toti^a quo; the - 
perpetuation of divisions and distinctions that are no Ibnge^^ valid 
or that are inappropriate for certain studies; and the probable 
undervaluation of basic research activities. , 

Multiple Taxonomies • Most organizations employ more 'than one 
taxonoioy. Generally they are employed in cons.tructljig the successive 
aiibdiyislons in a hierardty "ot suborganlzations: divicflons, branches, 
groups, etc* Thus, within a Division of Hij^e^ Education (defined 
according to a client organization taxonoxqy) there mig^t be. an 
Adminlstratioh"^ Branch (defined according to a sid>Ject area functions 

iaxonomy), and within it, an Applied Research Group (defined accord- 

• • . - / _ 

Ing to' an R&D functions taxonomy). This Is the structure associated 

■■*...•' * .■ , . , •. ■ " . _ 

vith linear lines of authority. The result of such successive 
subdividing appears to be extensive specialization and sjtrong 
communication tip and down the structure, but poor coio^ni cation and 
coordlna:tlon across /t. . V * 

. The several t^onpmies cfan also be employed' in constructing the 
two ovei;lapping subdivisions preerent in a matrta: of suborganizations: 
department^ and projects, branches and segments, or division^ and 
.prograsis. Thus , ^taff may belong to a Biodiemlstry Department 
(defined according to a discipline taxonomy) and to a Cancer Chemo'thetapy 
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Project (defined accojrdlng to a problem taxonosQr). This is the 
structure associated with matrix lines of Authority. The result of 
such overalpped subdividing appears to be Improved pommunicatlbn' and. 
coordinatlx)n across the organization* 

* Roles and Functions 

IWo contfastlng approaches t^b the asslgnmentV^f^V^'c^ti^iaion 
rolee and functione exist: functional comprehensiveness and functional 
specialization. • , , * 

Coi»q)rehensive , I Subdivisions are functionally comprehensive if 
they may carry out 411 (or almost all) of the institute's functions, 
but only vlthlii a prescribed portion of the institute's area of 
responsibility. The NIH Instltutef v some NIMH Centers, and NSF 
Divisions approach functional comprehensiveness in this senses 

Functional comprehensiveness has the acbantage of enabling all 

the organlza^on's. tools to be fopused on one important' area of 

an organization's responsibilities. Its dieadvantix0e. is its*" tendency 

^ to "lock-In," by institutionalizing i concern for that drea of the 

• . , - . • • - • . ^ 

organization's responsibilities. Functionally comprehensive sub«* 

divisions ""of an instJ^tute acquire special eiq>htois when they are 

calied '-Center" or "Institute." , t 

Speclalizedv Subdivisions are functionally specialized if they 
, carry out only;,a specified part of the institute's functions, but 
do so for the institute's entire area of respbnsiblllty* The NIMH 
Divisions and USOA's Services approach ftmctional specialization in 
this sense. . ^ \t ■ 

The choice between comprehensiveness and specialization is 
clearly determined by the subdivision taxohony employed in defining 
- subdivisions. For an H&D instittitioni the R&D Functions taxonoimy 
leads to functiiot^al specializatlph; most of the other taxonomies 
suggest functionial comprehensiveness. . • * 

Functional specialization has the adoantage that it permits 
#^ staffing, manageitaent procedures, and con^ituency contact^ to be 
specialized to the function at hand. Its disadvantage iB the * 
difficulty, in coordinating several functions (research^ service, and 



training, for exaxople) to solve specific problems and the tendency 

for functions (such as training) to be perpetuated with too little 

• ' " ■ ■ 

sensitivity to the changing liq>ortance of the task, ' 

MDffl Experience , The NIMH has both functionally specialized 
Divisions {Service, jteaearch, and Training) and functionally cosopre- 
henslve Special Mental Health Programs Centers (Crime and Delinquency, 
Metropolitan Studj^es, Minority Studies) • 

* There were a nunber of reasons for establishing Centers at NIMH,\ 
Ihese Include: 

o They are a means of attacking contei]^orai7 pxo^leiaa-^^ ^ 

that are not being add^ssed In the traditional 

functionally specialized structures • 
o They provide visibility (t:o Congress and other 

constituents) for what Is -being done in a problem 

area. 

^ • . •» . - • . ' 

o They provide leadisrshlp (to scientists and prac- 
titioners) fo.r previously dispersed activities in 4 
a field. 

- o . They stake outvthe, organization's claim in a problem 

area of potential interest elsewhere. 

o They provide an opetational base for measuring the. 

magnitude and importance of . a p^blem (through * 

Congressional administrative anjl constituent . 7 

' • ■ - ■ ' 

response^ to the neM^ly-^ylsible center) . 

, o They make it- easier ta -get additional liaoney because 

of their targetjjAg on problemiB . 

NIMH's expjerlence is that It Is iiq>o8slble to start a Center full- 

■ ' ' ' •* 

blown: if the Center* a Interest Is cross-cutting, thjsre may be 
little or iio Initial expertise to plan the prograc; too rapid 
growth may lead it into conflict for resources with ^ other Centers 
and^DivlBlons. Consequently, Centers are starred, small and gtow 
if Interest and findings warrant. The big factor in growth is — 
docs a constituency foria? , / 



The growth of an MIKH Center 'gpes t:hrough three distinct stages: 



o Coordinating Center, l^ls is' a paper organization 

. with no iDoney or control oyei;; xDoney. Its principal 

^ f - . . . • *. ' ' 

^ role is to 8tiiiRilat:e researdi in p;riority areas. * 
^ Its activities include Jouriial publication » 
conference sponsorship^ visits ^ and so on. 
o Limited'-funded Center. This la an organization 
vlt^ funds for 'res6|rdi on a narrowly-defined 
prol^lem. It could be used to test the research 
^■■r wmmmLty^a Interest in and acceptance of a particu- 
lar problem focus. No centers in this category 
exist at present* " >♦ 

^o Rilly^funded Center. These centers have fund& for 
research I training, and service and their own review 
comnlttees.. Since they have- available to them all 
the mechanisms of stqpport availablf^ to NIHB> j^hey 
are really mini-institutes. ^c^^y -ndscUi their dwn 
review cosnlttees » they can avoid the rlgiditiea. 
and discipline bias of cqnventidnal revieW' commit- . 
tees; they cun name non-professional ^ practice- 
oriented i&embers. 
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Most centers have spun off from the services portion of NIMH; 
when created V they took bveir the related research projects in the 
researdi portion. . Oi^e of the first centers established (in 1966) 
was that on alcohol abuse and alcoholism; it beg^ as a r^earch- 
only » limited- funded center. However » a a trimg constituency;^ « 
developed iiand it has since evolved to Institute status/uithin NI^. 

The disadvantages of NIMH Centers « ftomSthe perspective of the 
functional divisions » are: ^ . 

o They may lead to a segmentation of the research 

■ . . ' ■ ■ • , . . • ■ , r 

effort. * 
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j;^^-* o Their reseerdi nay be lec^e broadly ^nceived thaa 
that In a research divlaicm. 
o Bie appj^lcations of research findings to^ other 
- . progtans may be lost; 

• • • 

HOW SHOULD CONDUCT QF>R&D BE' ORGANIZED? 

Almost all Federal K&D agencies sponsor extramural research; 
* some also conduct research Intraaurally* There are thj^e principal 
issme concerning an agency's IntraiDutal S&D program: 

L ^ ' ■ ' • 

o what are the cdtfantagee ani diBodvantagee of haying 

an intramural program? / * 
o Hoy shotild an intramural program be orgcotuzed? 
o How can inimsmpal and extrmurM reeiearch be- 

interrelated? 

Th6re are intramural research programs at NIH» NIMH, Natlonttl Cancer 
Institute (NCI) , MASA-Goddatd, tisDA Agricultural Research Service, 
and OEO PR&E. The structural' characteristics of these progroas are 
summarized in Table 1. , ' 

Advantage s and Plsadvantages of Intramural Besearch 

- The prospective advantages and' disadvantages of an intramural 
research program are as foUovSs 

The potential adfuant^sd of an IntraiAural program to an institute 
include: ^ 

o Provision of an initial 6ftajc<a»s through vhitc^ a 

new agen<^ <can gain recognitipne 
o Creation of a model for reB&ardh^ especially in ^ . 

new areas and the, catalysj^s of research in ncv arease 
o /0>nduct of rtafc 2^as(^ ^ V 

. could not be carried out in extrasniral ins titutlotts » ^ 
o Conduct oi Icfig^tem Btudiea (e.g. , longitudinal : 
or epld€Smiolo^qal studies) for which continual * 
.fimding Is required. 
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o JtepHcdHcn of reaecordt canrled out at extrasural 

Institutions. ; ^ ^ 

o * BeveLopnent of neu Teeeco^ck tedtnP^uee too e^en«» 
8ive or intricate to be. developed by existing 
. extraniiral agencies » » 
.0 Faster ocntduat of cerrtain projects, 
o Creation of a critical mase of reeearcherB in areas 

mot supported in extramural aj^encdes. 
o Attraction md demtcpmBttt of goc^ reaegrch pesiaon^ 
^ naX Whp 'cioi jaove on to extraimiral research nftiage^ 

aent or institute administrAion. 

* ■ 

•0^ ikms^luct of lo^^-acoZ^^ complicated reaearch program, 
p Fro^d8ion<lf the strong in^hcuae ccnpetency needed 

to manage conti^aot »raearaft effectively. Without t 
such competence 9 it is iaoposslble to keep on top ^ ' 
"V ' * of the 1pi^ogr|aft on a dfior-^to-day basis « to evaluate 
' progress » and to recogni.7e and exploit advances. 
. o Eeady source of unUaaed infomation on BSD ptogtasi^ 

for institutift nanageaent*^ 
b , Source of higjhly trained RSD' ataff for extramral \ , 
c^V' inatitutiona ^- ^ ' ^ 

o Provide large» respected g]^9Up to proteut and \ 
prevent cppHcuttcn of poHticaUcriieria to\ 
scientific decisions^ * A 

o : Source of knoi^ledge to '^e used in satisfying ne?ds 

lot ahoxi^'-t^ns poHqf i^lev^ 
o Creation of aa intellectually challenging environ^ 
nisni^ to help in attracting fii^^ 
*^tlve staff. 

It should he evident thac i)pt all intraoural' R&D programs provide 
all these benefits to all agencies.; 

The diaadoahtagea of aigi intranural prograsi tp an instituti# 
include;/ 



o . Intiximral program tend to perpetuate themelvee 
: ^because of the desire of intraxmiral researchers 
to preserve their expertise 9 tlie formation of in- 
house lobbies t the need to maintain Intramural 
program lev^s to prese^rve productive research^ 
and the tenure of in-house staff in the civil 
service system. 

o The possibility of oonftiot with extranmral reaeixrckera 
who may, resent the relative freedom and stability of . 
intramural researchers and fear their coiqpetitlon for 
c budget resources. ^ ^ 
/ o ^ -The prospect that too much, of an institute's odbtn- 
iatrabvoe re$ourqeaj wilt be (iivdrted in' the early ^ 
stages of aii institute's growth. 

o The possibility that «too many ^aecrche/^B will be ' 

attraoted ftom epstramrfiL inetituHom where their 

' \ ■ • ^- ■ • ' ■ 

skiUs might be/better eiiq>loyGd to build research 

programs and train new researchers. ^ ' 

Organigatlon of: Intramural Reseaicch v 

The organization of intramu^tal research must be detetmlned 
by decisions similar to those discussed in conjunction with extra- 
mural research. This is 43b\dLously true when intrasairal and 
extramural are integrated. Separ^cte 'intramural organizations are 
ordinarily subdivided^ by discipline > occasionaljly by problem, .and 
in the case of the ARS, by subject «irea functions and commodities. 

Helationship of Intramural and Extramural Research 

As noted on pp. 19-23, the relationship of intramural and 
extramural research appears to depend jcritlQaxly Upon the type of 
research's the funding mechanism* used (grant or contract), the nature 
of the external research community , and the ^ incentive system at 
the R&p agency. NIH (most institutes) and NIMH are institutions 
at^ which-rlfltramural and extramural research are quite eeparate; 
NCI and NASi^Poddard are institutions at which intramural and 
extramural resea^^ are highJLy intei^related. 



IdU md NIMH Intramural research Is primarily basic biomedical 
research; their extramural re&earch is primarily unsolicited project 
proposals funded by grants awarded on the advice of peer-review 
panels; the extr«^ural community ^is primarily discipline-oriented 
^and universi^tyrbased; and the intr^ural reward system Is based 
on recognition by the extramuraiL^^wnity • a result, there 
is an implicit scientific compe^^HI between the intramira^r staff 
and their extramural peers » which underlies the separation of the 
intramural research and extramural research laanagement functipns 
at the institutes. Other impediments to cooperation come fromi : 

■ ► , ■ ■ ■ 

% • o the geographioal eeparation of Intramural and 
, extramural activities . S • 

o the eeparation at the NIB and NIMH Director^ a Office 

ley^Bls of intramural and extramural research 
o the competition for funds hetween intramural and 

extramural' research 
o the view th^t extramural propoaale are privileged 

conmmicationa that should not be shown to intira- 

mural researchers because they are 'scientific 

competitors of the ext ramural researchers 
o the tendency of intvamta^l reaearehera to become and 

remain highly apeddlized in research interests 
o t\iB dependence of intramral promotion on bench 
. aucceaa^ not interchange with extramural research 

Inanagers _ * 
o the fear that intramural directors would divert . 

rtoo inuch Hme to extramural programs^ if given 

such' authority 
a the experience that the peraonalitiea and tasks of 

intran^ral bench reaearehera and extramural r^isearch 

managers appear to be diffjarent^ so that the same 

person cannot function in both roles at the same 

time, nor can the twcT types fit comfortably in the 

same organization 



o the distaste of Intramural bench scientists for ^ 
working for "paper pushers" 

The one area in which intramural and extramu|:al activities come 
together at ^UlR is- oollaborative reaeca^dh. These are otdinarlly 
product-oriented 9 contract-funded activities that extend intramural 
, Interestvln directions that cannot be pursuedf in-house. As such». 
there Is considerably more Interaction bletween intramural and 
collaborative contract research than between intramural and extra- 
mural, grant research* 

NASA and NCI intramural R&D is disciplinary; their extramural 
R&D coiq>rl8es programs with specified goals and finite time-span 
consisting primarily of solicited projects funded by ciontracts 

, . ^ . . ' 

(in NCI*s case» this Is formally collaborative research) managed 

,/ • • ■ * . _ 

by staff who either fonneriy were in the^'intramural program or 
are simultaneously in it; the extramural community is problem- or, 
project-oriented and inidustry-based; and the intramural reward 
system is tied, to program and project success; As ^ result, there 
la an intimate oollciborabidn between intramiral and extramural' 
Teeeardhera in the adiievement of a apeaific goal which underlies 
the Integration of intramural and extramural programs at the ^ 
agencies. Other aids to :cooperation come from: ^ 

the use of a maprix form of organization^ tn which 
many staff members belong Both to a discipline- re- 
structured intramural ^ubgrOun and to a>i«Tobl^m- . 
structured extramural subgroup / 
the geographic oolooation of Intramural research 
ah4 extramural research management activities 

the provision of extramural funds to b0: used to 

• " " . ' • * ' ' 

support intramural research (to he2p entice good 

researchers to exttamural management) ' : 

the use of a consciousT staff development policy ^ 

that moves staff from IntraiSAiral research to * 

extramural program management . . 




the focus 01^ BSD areas (such as spaceflight and ^ 
cancer research) in which team effort is essential 
and project nanagesient provides challenges and 
satlisf actions . sufficient to attract hljghly- 
competent personnel (project iDanageinev:^ Is active 
father than passive, as In grants 'management) 
the tieed to rely on iritrantural staff , for progeot 
definition cptd propoiBal Selection because of the 
ln£^asl|>liity ^of peef^teview of Indtistrlal RiSD 



HOW SHOULD ADlfaNISTRATIVE SEkVICES BE ORGAMIZED ? 

■ ' ' " ■ . . • ■ ■ . ■ . • ^ . , /I " 

iCdmlhlstratlve functions include: 



o personnel services / ^ 

o finahciai management and budge*tlng ' 
o /fiscal services (voucher. auditing, payroll, recotd 

• keeping) > • . . ' 

o , management information and' analysis 

grants and contractis management ; 
o ""conference management (a distinct function at 

' nttH and NIMH) . * ^ 

• ^- » ' ■ - . . ^ ,^ . « "... 

These functions mig^t occupy about twenty prof essionkls eai*ly in an 
Instltute^a lifetime, growing to perhaps fifty vhen the institute v 
re^che^ 400 or so professionals (baseid Ton NSF experlencd) 

Centraligation Vcftrsus Deaatntrallzation 

The major decision to be made with respect £o" administrative 

■ ■ r-. ■ . 'V/- -■■ .... > ' ■ ' , • • *. 

functions 1? : how * much centrall2at:i or deceAtrallzation is - 

appropriate? 

. Centralization * Ifi a centrialized arrangement, all admlhistra-- 
tiveVpersonnel are in a single adiidi^ organization, generally 

reporting t^ the of fi director. Thiis has the aHvanta^e of I 

making It easier to apply the generally, large nuiniber of policies 
and procedures, consistently and, therefore, to ayoid'^ the situation 



in which grahteesi contractors, and prospectivi&^eKopl^ees ^celve 
different guidance from different parts 'of the organization. 
Another advantage is that such an arrangement enhances the prestige., 
of administrative personnel, petmitd higher salaries to be paid 
because of broader responsibility, and opens career adv^cement 
paths id. thin administration; consequently, the recruitment Of . 

0, 

good staff is easier. ' The disadvantage of this arrangement is the : 

tendencjr for separate administrative istaffs to become rigid in the 

application of administrative regul|itions and relatively unrespotisive 

to the needs of the R&D program staff • 

Decentralization . In a decentralized arrangeioent, admlnlstra* 

tive personnel are distributed; among the otganization's subdivisions, 

reporting to each subdivision Erector.- The advantage of this 

• • ' * ' ' ■ . 

arrangement is the greater responsiveness to the needs of the program 

' ■ ■ .. . ' . ■. . . ' . • • 

staff that should restjlt from closer association. Its disadvantages 

are the administrative ihconslQtencles^asiong subdlvlsl 

inay occur to the confusion of those who deal with more than one; 

the greater diffici^ty in recr^iting administrative officers because 

of the absence of a dear. Career path; and the in£d>ility to have 

much staff apecializatlon. . 

Combination ; NASA'-Goddard has developed an effective coiiq[>romlse 

solution to the centrellzed/decentralized choice ~ a xonbined 

arrangement, that f its^well with the matrix management structure 

employed at Goddard. The administrative division is treated 

Just like a disciplinary department when a new spacecraft project 

team is\ fottaed. Each project has an administrative officer assigned 

to the project manager on either a solid or dottedrllne basils. Each 

project team has its own coiq>lement of administrative personnel tiror 

budget, contract servlceis, and so pp. These personnel ar^' generally 

solid line to the project* The purpose of this a^aslgiiment is to 

give the Administrative staff an understanding d£ the project and 

a sense of participation in it. Their role is to facilitate Ibhe / 

proJect^s' goals by helping the project managez: do what he wants to 

do. Their rewards are linked to the projectfe succ&ss. At the *; 

same time, their- association with the .central administrative staff 



-39- 



and their return there between projects provides them with an 
attractive career path » access to b^ack-up specialists, and a source 
of consistent guidance. They also keep central administration 
Informed of the project's progress. 

WHAT PERSONNEL AUTHORITIES ARE DESIRABLE? v 

Thot basic personnel* authority \mder vhlch Federal B&D agencies 
operate is the Civil Service System^ This provides for: 

' ' ■ ■ ■ ■ 

o jobs to be. filled through a ^coiiq>e tit ion asong 

qualified applicants* » 
o approval of job offers by the Civil Service Commis- 

sion ' 

^ • . ■ • ' 

o . rank (GS-1 to GS-18) to be determined by Job 

■ ^ ■ ^ .■'•>.-. . 

.responsibilities, not qualifications of 'applicant 

o Salary to* be determined by raink and step within 

■ ' * ' . ■ ' ^ 

^' V. rank; new hires, start at the first step V . 

^ ■ * ■■ . " ■ ■, , • 

o job tenure to be obt'ained> after one year of 

probatilon « ^ 
b the total nixnber of Federal 0upergrade (GS 16-18) 
ly positions to. be fixed/ and distribution to agencies 
determined by a bargaining process - 
o professional and technical positions requiring V 
skllle in areas such as medicine^, physical 
science, and engineering are not s.ubjett to the 
. llmltaitlon /on number of 8upergrade8^ but posltiot^ 
requlrifig^ skills in SQclal or behavioral science « 
"are. ' ' ' ' 

. .■■ ; ' • ■ ■ ■ ' » ■ ■ ■ ^ ■ ' ■ 

The Civil Service system was designed to satisfy^, the need for a 
stable 9 conventional bureaucracy/ insulated f rOBOL polltipal^nf lue'^ce 
. and open to competitive advanteffent on the bas^s of competence. • 
It w]as not designed with the needs o£ R&D management in mind and 
.does not serve them £o well. Its disadvantages in this regard 
"V. ^-are-.that:. ' '■:'",\,\ 



o It ei^haslzes stability ana tenure^ when R&D manage-"^ 
sent organizations need a high degree of staff turn- 
over In ordef* to bring In personnel who have fres)i 
knowledge and Ideas . V 

o It enqfihaslzes rank as a function qf^Job description^ 
when th&^ competence and value of a scientist Is < 
determined by qualities not easily expres3ed In a 
concrete job description 
o It emphasizes equality of trea^nt with regard to 

salat^es and prosiotloti/ when most successful S&Q 
. ^organizationis find that strdng rewards for achieve- 
ment are needed to attract and« retain hl^ly competent 
staff V ^ - 

o It makes the firing of unproductive staff exception- . 
ally difficult and gives them little Incentive 
continually to upgrade and refresh tii^lr skills, as 
they must If they dre to serve rapidly advancing R&D 
■ areas • ' ' . ' 
o It recognizes the need to provide competitive 

salaries for physical and medical scientists , but 
not' for behavforal or social scientists or highly 
qualified p^ractltloners ^ * . 

o it luposes delays that make it difficult to make 
timely job of f erS; to I>r6fesslpnal8 when they are 

* available/ 

* '■ ' . ' , ' ' ' 

The Congress^ has recognized the liiapproprlateness of the 
conventional Clv;Ll Service personnel system for. management* of R^ 
and^^has given cettaln R&D agencies/ NSF and NIH in partlciilar, 
spet^ar peroprtheZ outTioKii^d* These are unique to each agency « 
On the baisls of J:hat experience » the provisions that a ispecial R&D 
agency personnel authority might have aires 

V'\-o': PiHng / _ 

y The ability tcf] hire 'qualified professional and 



technical personnel without a formal competition 
«and without having to wait 3-4' months for Ci'Hl * 
Service Commission approval 
Tenure and Turnover > 

The , ability to «limi t^ tenure to short periods » 
renewable only, on demons ti :lon of continued 
C0Dq>etence and productivity. A portion of the 
st^tff should be hired for fixed, terms without 
tlhure to encourage a constant rate of staff tum<- 
ovexi "^(fi.t should be observed that NIH administra- 
tion believes tttat the existence of long-»term 
Civil Service tentOre is essenti^ to NIH 
recruiting in competition with j^enore-sifferlng 
universities.) y 
GracleB and SupergradBB 

•The ability to determine grade and step by the 
salary a professional staff member can pbtain^froin 
industry or the' university and not by some arti-^ 
ficially^deflned job statement. There should be 
no limitation on supergrades for scientific and 
professiohai employees except the requirement that 
6ach person^s grade and step be justified by market 
factors and perfbrmancje* The inequity In^^t^ 
of biBhavioral and social scientists should be 
removed/' - ' ^■ 

Salariee and Promotion ^ 
The ability to tie salary increments and promotions 
to perfon^ce and not' solely to seniority and job . 
descriptioh. It should be possible to pay a within' 
grade salary to a .newly«*hlred staff menber if that 
corresponds to his market value. 



o Leaves 

The ability to provide each scientific and technical 
professional employee with a sabbatical leave during, 
vhich he nust refresh hia^^osipetence through active 
research, teaching, or studying. (NIH experience is 
that people who go of f on leave bring back good 
ideas.) ^ 
o Firing ' . ■ \ 

The ability to vewov^ scientific and technical 
' professionals if their competence and performance 
fail to keep pace vith^ the n^eeds of the organiza- 
tion. ^ 

WHAT PERSONNEL QlftvLITIgS ARE DESIRABLE? 

We qualities required in successful staff mesi>ez8 will 
differ among: / . / . 

o esottcammit program managers . 
o intrloffnuml researchers 

o- adniniatrative staff i 



Extramural Managers 

The qualltiles required of extrmtural program mmtageris .differ 
according to the nature of their program management role, i^hich- can » 
fall within a* vide range, bounded at one extreme by the Job of the 
NIH Executive Secretazy - and at the other by the NXSA*-G6ddard Project 
Manager, * * 

\ o hi NIH Executive Secretary manages the prQce8| of 
• review of unsolicited grant proposals by a study 
section coiq> rising ndn«-NIH scientists in a specified 
discipline. His principal role Is to engineer the ' 
applications through the review process* 
o. An NSF Program Officer is typically more aggressive 
\ in the encouragement and discouragement: of proposals^' 
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in the selection pxpposal reviewers » and in V 
aaklng'the final decision, 
o A NASA Project' Director mist determitfe what R&D^ 
is necessary » solicit its performance » award 
^dhtracts* monitor their perfomanceV and coobine 
them to achieve his project objectives. 

e . ■ ' . • • 

At NIH most executive secretaries and ext|^aimiral program officers 
(within the Institutes) \come from research backgrounds. Ablut 90;C \^ 
have some research ei^erlencey 902 hai^ a 1^.^. or M.T>. Many pass 
through a training program for ^science managers run, by NtlS. Fewer 
than lbZ> howevdr^ >coma from NIH's own intramural program^ where 
tihe- "publish or perish" syndrome discourages scientists from entering 
management. Attempts at NIH i:o train administrators ^ be. good 
project of fleers- have failed due to'thelr lack of science background. 
Once a scientist takes on a management task he qidckly loses his 
scientific competence and in NiIh and NSF eaqperlence it has been * 
impossible for him to get ^ back. . Thus » the movement is one-vay 
from, research to management^ " the aceverse is ixoposslble. The tu^o^^ 
of executive aecretarief is ismalt* ^V" 

\ «The Institutes have foimd tW 2«"year Public 
Officers make' good cd7Xd2^c)iMirtiva^ research managers.^ They know the ^ 
8clence> but not prbject man«^ement| but are generally willing to 
dig in and learn; ^lle; their older > more established colleagues 
are not willing to do*so. 

# 'At ilsF almost every prog^^ 
but he is made ai^are upon b>ing hired that his Job is management not 
..science^ Typically NSF program officers are young man on the. way up 
or older men who have left research . Th^r administrative experience 
and skills make them prime targets for university recruitment.' Many 
lekve the Foundation for college and university adidLnistratlve Joba 
department head > provost r college president. MSF has many 2-year^ 
jrotailng program of flters (especially In the blosdepces) . M^oty . 
program officers are hlrbd on a leave«-of-absence basis so that ^ 
they dan have a safe retreaV. > 



At NASA^Goddard a project director nay be: 



^ the study manager vho devel^iped the program plan 
o the Goddard manager |drfao has the best technical 

competence in the program's crucial area . 
o the manager of a successful terminating project 



The. first prloi^ty In choosing a project director Is his technical 
competence; che second Is his management esqierlence. If the 
project director do^ not have hl^ cos^etence In a crucial area» 
his assistant must* NASA-'Goddard develops most of Its own project 
directors f rom vlthln Its Intrasmral staff; a staff development 
program explicitly arranges for a nuab^ of years^ of Intramural R&D 
followed by extramural project management. ' Hew project directors 
are trained In'^a project, management simulation called GREHEX.^ 



Intramural Researdiers : ' 

Thift qualities required of Intramural xeseKM^etB are 
thoBe of competent sdentlsts anywhere « It Is useful to noke that 
the quality of^ NIH's intramural, program Is ascribed by some to its 
considerable rate of turnover and hlg|^ selectivity in making 
permanent appointments/^' 1^ draft deferment afforded FHS Felliiws 

^has asi^r^d tjti^ NIH a f JLow of the best young blosclmitlsts for 
two-year tems.^ The best among these are of fered permanent positions 

'In generals NIH appears to have been better at gr^q^flng its own 
researchers than at recruiting from outside at a high level. In 
addition to the PHS Fellows » NIB has a large nuni>er of other short- 
term appointment prdgraiib ~ V 

.Foreign Fellow^, e to/ All of these serve continually to refresh the 
scientific coiq>etence of the institutes • J 



McGregor, *£• B* and R. f . Bakers "GRE(SX-A M^pagemeiit Game . 
for the New Public Administration/' i^Ho A^nietrixtim ReHe^ 
Vol. XXXIIt Nb» 1, January-Februdry 1972» pp. 24-32 
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Ihe ARS has an explicit policy of providing parallel career 
advancement paths in research ana research administration. This has 
nade it no longer necessary to go into research administration to 
obtain a hi^ salary/ The career guide given to. each new eioployee^ 
vfalch explains this system, *^is considered, by ABS to be its most 
i.Ap6rtanC piece of paper. 

AdmJ ^n iBtrative Staff , 

the qualities required of administrative staf f > especially 
lg;rants and contracts officersi besides the natural ^administrative 
competencies, are two: . I 

o the .understanding, that it is not their role to 

make* sclenjcific judgments 
o die appreclatl<m that it i^jytielr role to show 

the program officer hov to do what he wants to 
* ■ _ , ^do. ■ * " ^ 

HOW ^ ADVIsblOr COUNCILS BE ADMIHISIERED EgjfecTIVELY? 

There are three ^oataiaauea in -the selactlon and effective use 
of an advisory council: 1 ^ . 

\ " " ■ - 

o What inopoaaa stiould t^^ 
V o How should its TRambeiv^p be detern^ ^ t - 

o How sjbould it^perata? . 

Puyose^ 

The purpose of an adids<yry council can be to provide advice or 
assistance in one or more of four jprlncipal areas: - 

o progriuB objectives and priorities <v' 

ot program substance v -v • 

. • ■ ^<.. ' 

o >pr9ject selection 
' o political and constituency suppjort 



Program Objectives and Prlorltlea ^ Advice on program objectives 
and priorities ^is probably the most common advisory council role« « 
When ti^e council is well-chosen for the prestige 'and representative- 
ness o£ its members, the council's ImpidLmatur on a program serves 
to yalidate and justify what must in the end be subjective, value- 
laden decisions » essentially political in^ nature. Some coundla 
'are chosen because their ideas and judgments concerning objectives 
and priorities are truly sou^t by the agency; in that case, 
extensive supporting stiift^work and frequent meetings are essential 
to provide the Information they need if vise advice is to result * 
from their deliberations. However » many councils are chosen only td 
provide the proper imi^rimatur tov^bjectives a£d prlorLtieB detemdhed . ^ 
by the agency itself; In that case, «no staff support and short, 
busy meetings are the rule. 

' Program Substance . Advice on program substance is another 
common role for advisory councils, especially when they oonq[>rlse ^ 
prominent scientists^ Some councils serve as usefiH. channels for 
ideas from the agqn<^* 8 constituencies.- (For eicaioiiple, NIH institute 
advisory councils channel ideas from organizations sucfi as the 
American Cancer Sci^ciety, hospitals, and medical schools.) Some 
councils provide continuing access for the agency* to members with 
expertise that could not be afforded or obtained in the full-time 
staff. They provide fresh ideas and suggest new dir^^ctions for the 
ag^tncy's' programs. ^ " - , * 

Project Selection . Assistance in project ^selection is a less 

sequent role of advisory councils, although the NIH institute. 

' ^ \ " •» ' ' . ' ' . ' ' ' * 

councils all have that responsibility formally. Their principal value In < 

this role^ lies in their ability to give legitimacy to necessarily 

arbitrary decisions; they insulate the- agency froqi certain^kinds 

of poll tical\ Attack and in^licate to the scientific cominmity t;fiat 

the award process is not dominated by bureaucratic capriciousnlbss. 

' Political and Constituency Support. Every advisory council 

plays ar, important/ though of ten implicit, role in building political 

and <»>nstituency support for the agency. While few agencies go so 

far as to have their isenAcrs form.tlly lobby h^totQ Congress i advisory pi» 



council xoeober^ fz^quently testify before Congressional cdssiitteeS) 

hold informal discussions ^ith Congressiaen^, and engage in other 

restrained forms of advocacy at which they can often be n^re effective 

than agency staff* Their adwcacy may also go on vlthlja the Executive 

brancl^^oth within the departments and in the Executive Office' of 

th^ President. They ^lao play an extremely important role in building 

a professional constituency among their colleagues for the agencS^. 

♦ 

It is generally intended that they will "tell their cplleagues that 
the agency is on their side and doing a gobd job for the nation." 
Conversely y they c^ sensitive .the»^elUEy to |fir<^lems in the scientific 
and leyi communities', * ^ ^ 

\ The National Institute of Dental Research formed an advisory ^ 
council of prominent university scientists and engineers when It 
wanted to initiate a program of .dental research in universities 
outside of dental schools, the council ^as ^le to advise the 
Institute on the capabilities of various universities^ provide 
university contacts » and assure t;he uivLyers|ity comimmity of the \ 
serlq^ness. and challenge of dental research and the quality of the 
Institute* The Council played an liq>ortant role in building receptlv-* 
ity for the jprogram in the xuilversitles and in Identi^ing ^proprlate 
sites for research centers* - ^ 

Problem. There are a nuabf^r of problems >and differences of 
opinion concerning the prbi»er role of advisory coxinclls. 

A major problem concerns the appropriate degreo ^f authority 
for couacil^; that ls» the |>toper balance betwisen cuivioe^givinff and ^ 
dBciBicn-^mciking . Those who belie;ve that advlsbry councils should 
limit themselves to giving advice fear thiA if they are given 
^decislon-maklng power » control of the Agency will be turnedv^er " 
''to some! taction from the outside resear:ch conpipuntt^,' iKirther/soxaeV 
fear' that the research community as represented on t^e uSc^ advisory 
panels will be^ too tied tQ the atatua qiio to cht^ge ^hen th^ agency 
should^ *Those who ar£ti§,'ln 4favor of dedslbn^i^ijig po^fr believe ^ 
that such dMthority is necessary to attract and retain ^the activ^^ 
inteij^est of flr^^class menbers lEor^^ council. 



The difference of opinion Is^ actually iDore complex because It 
concerns whether or not advisory councils should have decision power 
vltfh regard to each of three areas: . 



o program objectives' and substance' 

o ^program sub^tapce ^ ^ - 

o |(»roject selection . y 

One vfev is that councils should have decision^ power with regard to 
programs^ but should not bd suboerged in the details of project 
^1i^l|$;tlon» The ^ opposite view holds that only by becoming familiar 
with l^e specifics of projects can council nesibers gain a sufficient: - 
undifet^^^nding of an agency *s program to be able«.to provide useful 
advi^. Part of the resolution of these differences lies with the 

-■C'"'iiifH&ts of staff support and pre-'neeting council efforts undertaken; 
with proper support and effort a council can exandne inany' projects 
without becoming overwhelmed by detail*^ Another ^art of the 
resclatlon depend^ on the seriousness of the council's independent 

. revlei/ of the decisions it makes; if it . is willing to examine in 
detnail only staff-^seleoted special cases,- it' can exercise decision 
authority withotft undue effort • 

' Another problem implicit in what has been said above is the 
balance bettieei^ the c^tH^Bor^ oomail^Sh direGtor^e 
pcwer. Some advisory coim tils run their the 
agents o.f the agency, director « This is only partially determined by 
formal assignments, of respondibillty; the x:haracter 

of the director the membership of th^$<3^clliv^^^^ way the council 
is supported and used. ^ < ' 

Membership - / : -■'-^ .. r ^*vvl-. ■ 

. 'The membership of an advisory t^^ujiifcilja^^ b^ 4^ermlned 
according to the purjkosea of the cbi||cii^^^^^^^^ 

It iflT intended to haye, and the ways ^ jp^^ supported 
aiid used* Naturally, council? Intei^^lp^ agency' 

deci^lons^^ to build constituency imd Coji^r^^^ and to 



prgvlcle broad program gufdtobe are con^rlaed of scientists » practl*- 
tlonei^d, and representative^ of the lay public chosen primarily for 
thelT emiiienbe and Influence • However » Cornells Intended to 
participate effectively Iniimaklng agency program and project decisions 
and to contribute useful substantive Ideas comprise members . « 

chosen for their ii0^1)apetence» willingness to devote considerable time 
to 'council abtlvltles, and k)lllty to contribute effectively at 
meetings. The balance anbng scientific , practitioner, and public 
. repVesentatives favors tie former where the agency's maj6r concern is 
with basic research^ and shifts toward th? ^lattfr whether the agency's 
major concern is with applied and problem-ox^ehted R&D. However, 
the effective -use of public meni^ers on NIH institute advisory 
councils had required special efforts to draw th^m out because lay 
m^ers do not feel competent to evaluate tae scientific projects 
and hesitate to comment on. the quality of science. 

NIH institute directors emphasized the Importance of the 

council members' personal qualities in making ^council meetings ^ ^ 

■ • _ ' ■ .wi^^ . ' ■ > ' ■ 

effective. They believe that the institute should: 

. . • ... 

o chdose council mendbers for style . and Interpersonal 

sklll'before professional competence,^' ^ - 
o choose members .who have a strong polnt-of-vlew. 

■ • • • ' . . . • '■ - ■ ; ^- \ * 

Politics plays a role in council member selectiollv since"^ nominees ^ 
are ordinarily approved at departmental or higher! l^vels^, but^ som^ 
aigencies feel that that has its benefits: political choices are 
believed to be better connected to funding sources an4 better able ^ 
therefore, to assure the agency a hearing for 4>ts programs and 
iud.get requests. An attempt is usually made toj^alance the repre<r . 
sen tat ion on advisory councils so that they Include: ,a range of 
scientific disciplines, institutional and geographic spread, 
representation of minority groups and women, and^ (isome times) ^a range 
of ages. ■ • . . , '* ' ' ■ _ '"".^ 

A frequent' recommendation was that advisoxy council, meml^rshlp 
have fixed time limits, otherwise there is a tendency for a small, 
fixed group to ioaintain dominance of the cQuncil for a long period. 



Operation * . 

The effective functioning of an advisory council Ib MgKly dependent 
%on the way In which the meetlrigs are organized, the staff work Is done, 
and the director participates. * - ^ 

Experience with the NIH institute advisory councils ciuegests that 
effectiveness is enhanced by the following procedures: 

o* *2%e iriBtitxite director aerves as counoil ahairman^ 
'^Slnce the general role of advisory counxd.l« is to 
establish two-way conmunlcatlon between the various 
constituencies and th& institute, it is necessary 
to have' someone in th^ chldr who knows 'the institute 
well; that '^is not likely to be a council menber. \ 
Moreover, xmle^s the director is chapman, a tension 
will be established between the director and the 
council that is likely to result ih his not bringing ^ , 

issues of consequence before the council. 

p Meetinga are arranged to permit aa much diecuaeion of 
ieeuee as poeeible. One Institute proceeds as follows:^ 
Before each meeting, the Institute staflE develbpsxhe 
key Issues fori disiussion. The director asks one 
member of^ the council to think about each issue and 
present his thoughts at the meetings. This leads ^ 
off the discussions; thie institute director then 
atteiiq>ts to focus the d^cusslon ^during the meeting.^ \' ' 

Another institute relies slightly liiore on staff ^ 
work before t4ie meeting. Once the specific topics ^ 
to be discussed have been identified, the staff 
prepares background papers on each one. the director 
selects one eoundil member as "Agenda Chairman" and ^ 
^ . goes over thei agenda with him before each'* meeting. 

During the meeting, the director: . . * 

^ o does not ask council members t6 directly 
^^te their positions on a policy issue. 



in the belief th;d[t nost of thei^ are 
r^ticoit about taking unequivocal stands 
. o lories to keep' debate from getting angry, 
/in the belief that dngry menbers will nbt 
spread a ^sitive view 6f the institute 
o tries to draw out the coimnents of lay 

oeoberst Vho are reticent in the presence * 
of scientists ' \ , 

^o seeks to have everyone participate aiad 
have a feeling ^ participation 
Still another institute insures i&ffective discussiona 
during the council meetings by dividing the council 
into subcommittees for discussion of the issues . 
before the general oQuticil meetings; the sttbcommit- 
tees then make recommendations to the general 
coundLl. t ) . 

Metrbera are%nvdlt)ed personally^ and informally in 
institute ixf 'fairB*^ The directbrvof oxie institute 
inakes a conscious effort to get to know every 
council member perisonally', especially through 
arranging social events. Between mdet|Lngtf^ he ; 
engager iti private personal communications with 
council members; he goes to them d^Jtoo for ifdvic^. ~ 
and involved them ih institute activities (for - 
example, having them head Up task forces or serve 
on committees); ■ i' ^ : ; 

The 6om(At is liBterie^d^ Fundamental to the ^ 
^effectiveness of an adviaory council, of course, , 
is that, a ^eellng!'x)f mutual tru^t 4hd respect" ' 
deWiop between tiiie counxill and;' the institute. 
6nly if the staf f ipl the ihstitute asks for and 
listens to the council's advice yill ita members * 
be thrilling to devotci ihe effort oeeded to provide . 



advice th at^ will be ~ lia tened-irpii- 



ni. kANAGEMENT DECISIONS 



WHAT'^SgOULD BE THE MAJOR miURES OF THE MANAGERIAL SmE? 

iour majpr featureB^ servex to define the basic management 
style of an R&D agency in the auppb^t of 'extramural research • Theise 
are; ' * ^-'y ■ ' / * 

' ■ . ' • ■ • . • ' ' •>■>■. . ■ ■ ,■ ■ • 

' . r U - * ■ ' .. • •* 

o The allocatioH of mnageHat effort among the 
■ i phases^of actiirLty/ ■ 

> The degree of oont^l exercised by'^the agency 
.> , over the REcD activities it siippbrts. 
^ '^o T^iB' extent of external Tpca^ v 
management process. ? ;i ^ 

^- o Vxe vconety of nmiagement ety eiq>iQyftl l>V:,thi& ; 
agency* ' 



0 



Allocation of ManageriTal- Effort 



R&P agencies ati^ e^brts in different waya 

ioBC^g! th^ three p.rlnclpal pha^es*of R&I) actl^^ plaxinltigt d«velop^ 

• ■ ■ : -. • '■ ■■ ■■-■■■^ ' .;;^-^-,v ■ ♦ -'ir-i^' -f • ^- 7:'^- ■ - 

ment^^ and evaluationvj ^vThes^ different a^lod^ti|t)n8V:d^ ^ 

' d^fferer^t philo&voph^e.r of R^^^ mSxsiz^ts^nt^ different R&D , . / 

objectives*:" , :^_ . /r:i^i-^:-^y' -^'^^ ' . -C'^"'-;^ '^'^''^-^^^v- ^■■'•^V 

. Development ETOhasis .p. Pnfe^ j^^ jj^ many R&D^'agekicies , 

has |)een to place |rimdt^^mat^^ 
fun4jLng of individiM proje^ 

R&^ activity. In this a^prbai^^.^ ea^ncit^ 
to the prior planning of ' cbmprefe^^^^^ or -tp tli^-'-^lvaltfati 

of program pei^fmahce- after^^ itiltKese^ 
cases ifli almost entirely cortcenite^^ thi^^rojecta 
to be funded from among a lalbg^e cbriection 
The basic management styles^ of NII^^ and NSF shii|^ thl^!\^ 
emphj^lsj they differ p^lnc^^^ the mechanism ecqildyed fbr 

* project ,4eieciionv-'|v^^ — -/'f^""' -'^ ^^S''^"'^ ' - '^ . ' , % 




The adoantagee of this eiophasls apply vitti jgrcfatest force when 
the R&D being funded is bast^ research In well-developed scientific 
disciplines. The development eiq>hasls has Its pHllosophlcal basis 
In the beliefs that: * 



o scientists at. the frontier are the best sourcesv 

y of Ideas^ about >Aat needs to be done; 
o wMdt^clentlsts will continually generate those 

/ideas and present them to the funding agency; 
o V selection among^ the proposals should be based 

primarily on the scientific q^uallty of the proposal 
and the competence of the proposer; 

o the best scientists, will not or cannot work on 

. • ■»'■■■■".•,.■ • ■ " . 

other people's ideas; 

o the best ^esb post evaluation of a scientist's per* 

* • formadce-is provided by his peers in the form of * 

the decision to publish his papers in professional 

journals and the reputation that h^^pbtalns. 

In sum, with this^ emphasis the R&D agency serves as a selective 
funding source for a more^^or-^less self-governing community of 
scientists; the agency Jhas only indirect influence over the topics 
and evaluation 6f the research. 

The diaadoantagee ot this emphasis are:- _ ^ 

o it;does hot sufficientiy serve the public interest 
. in assuring that public fundsi are expended to meet ; 

public tneeds; / *• ^ , * v ^ 

o it does no£ , permit sufficient coordination of . 

activities' cotiduct'ed as separate projects; 
o it does not piQT sufficient atC:ention to aissuring 

that the {Projects supported actually are well 

carried^ 0ut; , 
'o it relies too mu6h upon the self-direction and 

self-evaluation pf the R&D community and there 



are portions of thevJi&D community for which such 
""reliance would bje uiwarrantedit / * 

10 it is a^propria^ at beSjt for the support^ of basic 
^researcll; bub is inaj^raptiate for {practical and 
prc^ranmatic.K&D in whic» goals must be established,! 
< prcqjiraitis^^^lanpedy and prog^ress monitored* 

. Planiitng Enphasis ^ A second major approach places the primary 
management exi4>hasis on the design of |L&D jirograms comprising coordinated 
feries of projects; that Is.^ on the planning phase. The basic manage- ' 
men t styles of NASA-Goddard, AiT Force weapons system development « 
the National Cancer Institute's collaborative research, and many' 
industrial R&D agencies have this primary emphasis on planning. 

Ihe advantagea of this emphasis are strongest when the R&i> being 
funded *i8 product or process development eiq)loying well-defined 
clinical or industrial skills. The plianning emphasis has its *^ 
philosophical basis in the beliefs. that: 

< o the goals of an R&D progrw can be established 
"outside" the R&D community; 
o the iiecessary R&D project^ can be specified by 
individuals other than those who will be per** 
forming {he wolrk; ^ ^ 

, o the best way tQ assure achievement of the ^ ' 

specified goals is to design a coordinated .. . 

series of projects; ^-1-^1 „ 

o good scientists iemd developers will work on ^ 

' projects'. defined by others; 
o close external monitoring of project progress; 
is both feasible and desirable. 

In sum, with this emphasis , . the R&D agency serves as a purchasisr of 
carefully-defined services from the R&D community; the agency has 
direct influence ,over . the sjUb J eciL^ , . 
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The dieadvantagee of this einphasls derive In large measure 
from* fears that planning will extend to areas of research activity 
(especl^ly basic research) in which It mls^t be Inappropriate;' 
that the planning will be lncosq)etent and the monitoring onerous; 
and that creativity and quality will be Inhibited by too detailed 
a plan. . ^ ' 

However, there Is general agreement that for cpmplex R&D 
.activities in which there is a soj^d base of engim^erlng knowl^ge 
and technique., a nux(2>^r of coiq>6tent development organizations 

and we 11-^de fined go^als (such as In the"" design of spacecraft), a 

^ • ■ ■ * ' . . .] ' 

planning enphasls is appropriate.. The - primary disagreements 
concern^ the boundaries of propriety for such an approach: would it 
be suitable for managitig the design of new forms of education, for 

example?,. In what areas is there a sufficient base of knowledge, 

•. ■ <■-■■■■■■« 
technique, and institutions to enable adequate planning to be 
• * i. ' ' ■ ' .-■■■■? 

conducted? ^ 

' ■ c • . ... . ■ / . . 

Balanced Emphasis . A third approach places the management 

emphasifi^ in fairly even balance between the planning an^d development 
phases. Plantiing> while present, is carried out on a continuing, ^ 
incremental basis, rather than as a unique, coioprehenslve exercise. 
Consequently, the management effort being devoted to it at any 
instant is less than in the previous case. The development phase, 
Vhile it includes project monitg^ng as well as selection, shares 
the managerial eiqphasis with the )»lcaining actiyitles. / Thiis balanced 
eiq>ha8is is characte:iristic of sonu^ of the R&D managemeilt practices, 
of NIMH, the Department oi^ Agriculture, and the Of ficef of Economic 
Opportunity. \ 

The advatttagee of this balanced emphasis are strongest in the 
case of applied research o^ continuing problems of practical concern. 
The balanced emphasis has its philosophical basis in the beliefs .that 

o the goals of an R&D program can be established • * 
. Voutside" the R&D community; 
* o tXe continuity of concern for most 6f the agency's 



Issues means that planning may be accomplished in 
an evolutionary and incremental fashion; 



o good scientists and developers can be attracted 

to projects defined by others; 
o careful project selection procedures » relying 
iq)on practitioner as well as scientist judgmentt 
are necessary and* appropriate; 
o ! careful external monitoring of ^n**golng projects ' 
Is both deslr^le and feasible. • ^ 

^ -■ * . ' ' . ■ 

In sum, with, this eisphasis the K&D «tgency is linked with its R&D 

comminity in a continuing association in which the agency provides ; 

monitored support in ^return for the R&D community's dedication to 

the goals established by the agency. 

The' disadvantagee of tlt^ balanced emphasis derive from* the 
beli^^that it dot^s not go far enough toward either the plannl^^ 
or the development eiqpha^ls to satisfy the strong proponenjts oF 
* either; i - . .. 

Evaluation Emphasis . A forth approach is to place mtoagement 
emphasis on the evaluation phase of R&D management. To our knowledge » 
no agency has adopted lt» althbugh the Cooperative St%ate ^search 
Service of USDA has come close. Were it implemented, it would Involve 
. the devotion of co^^derable effort to reviewing the performanqe of 
projects aifd prograiad after their cosipletion; ^he resultant infor-* « 
mation would then be employed in the planning and development of new 
1^ programs and projects. ^ 

One ouipantage of duch an emphasis would be that much program 
plantiihg and-^ almost all project selection is accoiq>llshed n<^ on the 
basis of written proposal^ describing what will be done, yet in many 
cases this results in a premium on skillful writing and excessive ^ 
optimism concerning what might be acconq[>l|^hed by the proposed project 
or program. Were major efforts to be undertaken to evaluate perform- 
ance aftdr the fact, it would probably have a positive effect on the 
realism oiE^ proposed projects and programs "and on tjfie ability to judge 
the cOTipetenc<fii r>f prospective performers.. ^ ' 

Vx^ diaadvantagea of such an enqphMis ^1^^^ 
of actually maklnjg such evaluations (although they should not be 
any, more difficult to make than project selection decisions on the 
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basis of proposals) and of linking thek^ef fectivqly into planning 
and development decisions. ;^.r 

Degree of Ctontrol ^ 

R&B agencies have a choice of the degree of control ^hey will 
exercise over the R&D activities they sponsor. 

' Passive . At one extreme, an agency may adopt a passive mode, 
leaving the' planning, condyct, and monitoring of research iq> to the 
R&D comniunlty itself. This is the style generally adopted^hy 
agencies supportiilg ba^ic research. Its justification lies in the.* 
same aasumptlons about the R&D prt>cess that were listed in justifi- 
cation of a develqpmental eiq>hdsli3 for R&D management. Its legal ^ 
symbol is the gvcpit as»a form of support; grant*-dupported R&D is V 
.traditional^ly considered to be legally xmder the control* and responsl 
bil^ty of the grant recifULent. 

Active . At the other extreme, it may adopt an active mode ^ 
closely .specifying and moj^dtorihg what Is to Se done at each stage 
in the R&D activity. This is the style generally adopted }xy agencies 
'suqpporting development. Its justification derives from j^erience 
with engineering and applied research* Its legal syvbol is the 
contract as a form of support; contract<-^supported RiStD is considered 
to be legally subject to the control and monitoring of the contract-^ 
ing agency. , 

Intermediate , . In between, of course, die re are -a range of* 
balances between passive and active control that have been found 
by a nu]d>et of agencies. Some of : these ^oiint. to dianging\ the \^ 
degree df control at different phases of R&D management. For 
e9£a]q>le^. the agency sjay exercise close control over prdgr^m^^and 
project planning, but then spend less effort ii|;i aoonitoring; the 
progress of the resultant ;Rti) activities. * 

Degree of Jxtrgmal^articlpation 

Anol^er characteristic /that defines an R&D^ agency's management* 
- s tyle :ls" the degrei^" of ex^ in the 'i^erf ormai;ca 

^f its responsibilities. 



Considerable ^ NIH, for exmple^ ll» characterized by extensive ' 
partlc^lpatlon of th^ exteznal research community In derclslcmmaklng ^ 
during all phases oi^ management.^ Study sections » compzilslng 
proMnent extramural biomedical researchers, assign the basic ranking 
of propoaalis «f or support of extramural research; Institute advisory 
councils; cttop|;i;s^g researchers and practitioners, ratify study, 
section rjac^^ program Judgments. The NIH staff 

assigned tQ tiic^se^ of study 

sections and iiasti^^ — play a relatively, small 

r9le In the l^dLsl6nlD^ substantive' decisions are 

made by these l^temal:^-^^ ' • 

Minimal . (XIR.add NASA-^Goddard, however, place almost exclusive 
reliance on Internals st^f for decisionmaking. ONR progr^ officers 
rely on their own judgment to a greater ext«it thto basic .science 
program officers in most 'other agencies; NASA operates in ad R&D 
context in which relltoce' upon outside Judgmt'.nts in making contract 
awards would be Infeaslble and in which a hl^^ level of staff 
competence and awareness of agency needs reinforces tj^e inclination 
to reserve such judgments to, inslde^rS/ 

Determining factors . Several factors affect the degree of 
external involvement an age.ncy finds desirable • All else- remaining 
constant, the tendency to Involve "outls.iders will be greater: 

o as Involvement with b^lc research Increases (the 

outsiders will be basic scientistsi; 
o as involvement wltH.4evelopment Increases (the 

-outsiders will be pr^tltloners and users) ; 
o .the higher the quality and prestige of a field 
. of science being supported (the outsiders will 
4>^ s<^cntlsts in that field); ' . 
,0 'the smaller and less qualified ^the internal staff; 
o the. greater the need to build an external 
. constituency, for the agency; 
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the greater tlie diance of cgnfllct-qf^-lnterest 
for agency st^ff; o. \ 
the lower the chahce of confIlct**of-ltrtere8t ' 
for putslde partlcdlpaata. 



An agency supportlixg basic research in an area of high quality 
science and well-tralned scientists, which feels the need to build 
an external constituency >rill find it desirable and feasible to * 
engage nany external scientists in. its deliberations , as NIB has 
done. • - * 

An agency supporting hardware development .in an area oi^ high* 
quail tyv science and engineering, but strong coiq>etitive Indus^jjrlal 
interest, la likely to find it necessary to rely alnbst solely^ on 
internal management competence, as NASA and the Air Force 'have done« 



' Variety of Management Styles . V 

The final determinant of an agency *s has ic JR&l) man^igement 
paradigm is the number of management styles It uses in dealing with 
different R&D communities and objectives • 

' ^ NIH, for exanqple^ empldys its b'asic dual, revieu ^xocbbb almost 
exclusively (since it supports >aslc biomedical research almost 
'exclusively), whereas NIMH has added ^to dual review a variety of 
other management processes; 

coordiruxting centera to encourage other cosponents * 
of 4IIMI1 to iund ihterrelated research, service^ 
and. training programs in specified subject areas; 

o funded centere to supppi^ research, service, and 
training prognoas In specified subject areas; 

o tke mmtal health eeiviaea RSD^ program to * 

develop and implement ixapxov^A Ihodes of delivering 
health services* , ' 

... V ... ^ ? ^. * ...... 

Similarly , NAS^**Goddar4 employs one basic paradigm (with a nunber 
of internal vairlations) , while .the Department df Ajgriculture has 
added processes fo^ developing cooperative regions ^ we'earah pro^eate\ 
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to steet conoojft neetlft of several states^.and apeeial reaeardi projeate^ 

' . ^ ' . ......1. ' - . ' » 

to respond to hig^i^prlorlty iiatlonal probies^^ to its basic proems 

' for funding state agricultural experiottnt stations ."^ ^ 

-•^Miir A unitary, oanagement style » vhicit used easfthtiaily the sazae 

planning » deyelopnent» and evaluation processes for every H&D ' , 

activity supported by the agency^ has the adoantage of managerial 

^ 6iaq>Ucity and^ consistency; but it has the di%Spantage of pieoyiding 

inadequate response to variations in* the, R&D cononini^ies-and objectives 

thalt many agencies face. An agency » such as the ME,' tiiat Is likely 

to support a wide range^of REO) coommitles (from the*behavioral and • 

social sciences^o the physi:^ scienci^s and education) and a variety 

of R&D objective^ (frok basic research to developaeht and deiaonstra-* 

tion) will probably need to es^ldy a nuxdber of different management 

processes matched to the needs of its several R&D comnunlties and * 

obiectives* 

m ...•(.. 

ROW SHOULD BUPGEt ALIOCATIONS BE MADE? ^. . 

The primal nanagetient decision is budget allocation. The 
. manner in W|ich that allocation is made is -a crucial determltiant \ 
^ (f an R&D agency *8 management performance^ There are tho^e aapeote 
^ budget allocation, that must be cdnsidered in developing a 
management paradigm. They are: ' ^ 

6 By ^at proceae shall the decision be made?^ ^ 

o What budget atnioture should be e]q>loye(l .in 

decisionmaking? - 

' o * Wha^ factors llinlit or ekpand the flexibility 

' :' ' ' • •• . • - , • ' 

with which budgets can be allocated? 



Process 



1 



The process o£ budget allocation has three aspects: ita mode ^ 
cooqprehensive or incremental; its. direction — top-d6wn or bottom-up; 

and Ita pweedu)^ coUeglal or adverddTi.al« ^ *^ 

* - • • • . ' 

M6de # Theoretically*' at least » l^here are tv p opp o al Bg modes of 
budget decisionmaking: comprehensive and Inctemeutal. 



In the compreherwit^ vaode^ the entire budget Is assiused fo be 
available for dlstrlbu^lop In the way that maximizes the djgency's 
objectives. For a substantial R&I) agency^ ^comprehensive budget 
allocation every year Mould mean a considerable task of comparing 
enlarge number of possible programis against 'eaph other to select 
the optimal combination* But fomprebenslve budget decisionmaking 
g^erally occurs 9 if at all* only once in the Hie o/ ari fiU) agency ^ 
at its beginning. The progrann chosen then tend to develop a certain 
persistence titiat reduces thr flexibility of allocation possible In any 
Biibsequent budgeting period. ^Xi^nsequently, most actual bwlget 
allocation iQccurs in an inbremntat inode^ in which only a portion of 
tha agency's budget is really available for distribution to Increase 
service to the agency's objectives. This '.increment al mode of 
dec^slonmaklng^ubstantla^ly reduces the problems of making budget 
allocations (by reduc^^^ng the hunber of degrees of f reedpsO > while' 
also reducing (somewhatX the degt^ee to whi^ the resultant alloca-* 
tlon can optimize service to the agency's objectives in any one year* 

Those who favor cosprehensive modes of decisionmaking tend to 
do so on theoretical groundift ^ after al^l, comprehensive conipar^sons 
are the only way to assure that maximum benefit will be received 
from a given budget; while those who favor Incremental 'modes do - 
^ on both practical and theoretical grounds ~ it is unlikely that* 
ady agency with contimiing programs will have mre than a small 
pi>rtlon of its Resources truly available for reiallocatio)ti in any 
year and^^ moreover^ it is not clear that any R&D ^^ency's capacity 
to .examine alternative allocations is gpod enough to obtain the 
theoretical benefits of comprehensive allocation. All R&D agencies 
that we exADlned make cheir major budget allocations Incrementally*. 
fV \ Direction . Another distinction amon^ budget deciaionmaking 
processes concerns the direction in which the decision proceeds 
along linejB of authority: bottom-up or top-»downi 

^ In l>pttcam**i^p^6ci&£onmaklng, each of the agency program officer^ 
prepajpe's budget requests describing hle^ best judgment of his needs 
for the next;6udge^ period. His immediate i^upervlsor then Combines 
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-and modifies the budgets fron the seve'r^ progran officers reporting 

to him and forwards the merged budget to, his Immediate superior. The 

jgroceds continues up the line of authority until it reaches the 

'kg^*csy director who combines and modifies the budget^ to produce an 

' . • , , ' . 

agency budget that goep to the »respon:iible executive agencies and 

the*Congress. When che executive and. legislative brajnches act on 

the proposed budget » their decision leads f;o a modification of the - 

"^proposed budgets at each step down the line of authority. The 

"modification may be a simple ^qual percentage reduction (or ^expansion) 

up and down the line,* or it may be a different change at each step. 

In the latter, case 9 the process come| to resemble top-down decision-- 
> • 

making. - * 

In top^dotJn decr^lonaaking, the agency, director decides upon 
an allocation o£ the agency 's budget aioong each^of his principal 
subordinates. Each of them^ In tura> allocates his budget among 
hie subordinates*^ The procesis continties until each program 
officer receives- his allocation, which he then uses to support 
R&0 activit-les w , V / 

The difference be'^tween bottom«-up and tpp-down ^cisionmaking 



could be quite sObstanCial* If budget detislonmaking were done in' 
the comprehensive mode/ However, when the\ decisionmaking is 
Incremental 9 the two methods tend to merge into a mode in which 
signals- about the desirable and possible budget increments at each 
level travel up and down the line of authdrity. 

Procedure !. A third distinction among budget decisionmaking 
processes concerns the procedure by which the decisions are made: 
advajsariaj, or colleglal.' 

' In an advepaqrial procedure > the several subordinates compete 
for the budget allocated by their immediate superior; the amount 
eacli of the subordinate organizations receives depends on its 
ability to convince tfte superior 6i the value of its progra^ns* 

' In a collcgial procedure the several subordinates decide among 
themselves' how they wlil divide the budget ma^e available^, to them 
by their immediate superior; the amount each subordinate organization 
receives depends on its competence liji 'bargaining' with its fellows. 
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Both tolleglal and adversarial procedures are exq>loyed In 
fed*etal. R&D agencies* 

Strudture 

The^axononiy used to* define budget categories establishes the 
frauework within which the major resource allocation decisions at 
an R&D agency are laade. There are two isajor classes of taxonomy, 
leading to two alternative kinds of budget structure — Input- ' 
structured and output-structured. c 

Input-Structured . An ihput-structurgd budget Is one In which 

the radSoT Biib categories corriBspond to factors that enter the R&D 

process • These may ^e the Inputs recognized In conventional 

accounting categories — salaries/ facilities, equipment, travel, 

and so on; or the inputs defined ±n terms of R&D funding mechanism — 

basic research grants, development contracts, fellowships, and so 

on; or the Inputs defined "fn term^ of supp6rt.^for cert^fln groups of 

RSD performers (disciplines) — biochemistry, geh^^^f-cs, statistics 

and so on. While these categories differ considerably^ among them- ^ 

selves, they represent simply different groupings of the resources 

entering R&D activities. They provide data that is most useful in 

making administrative decisions. >lQ8t R&D agency budgets, are 

structured in one of these ways. Fdr example ,^ at NIH the budget 

is structured acpording to intramuiral, extramural, and special 

programsv and consequently, the competition for funds between 

Institutes takes place in each of these ^ categories .* 

Output-structured . An output-structured budget *is one in which 
t. » ■ • ^ ■ 

the major subcategories correspond to aspects of the results of 

the R&D process. These may be the resources associated with. 

activities serving each of the major agency objectives — develop 

leukemla^accine» design and launch oj^bitlng solar observatory, 

increase knowledge of DNA functioning, create new dental Amalgams, 

and so on; or the* resources devoted to progr^Uns serving particular " 

groups ^f users — 4alry farmers, feedgraln farmers, high school 

science teachers, leukemia pacients. The virtue of this form of ' 

budgeting is that It produces data directly useful in the process ^ 

of planning. By displaying the resources currently devoted to 



specified agency objectives or user groups, it establishes the basis . 

. " • ■ '* . . 

for rational consideration of. the reallocation of existing resources 

and allocation of new. resources among competing ageiicy. goals. As R&D 
agencies begin to devote, greater emphasis to the planning and evalua- 
tion aspects of S&D management , output structured budgets should gain 
wider, use. 

FlexibiUty 

• The budget allocation decision is affected by a variety of factors 
that operate in, one or another way to limit or expand the flexibility 
of budget choice. • 

Limit . Among the factors operating to limit budget flexibility 
are the following; • 

o Pvogram continuity. Most R&D programs and many 

projects hlEivQ^ great persistence. Some projects . > . 

simply require several years for^ completion; most 
UST and NIH grants are for *3 to 5' years for that 
reason. Others develop their own momentum through 
the growth of constituency groups; the 1<^F Summer 
Institutes' for teachers have significant constitur 
ency and Congressional support. . l^e consequence 
of these factors is that mature R&D^ agencies find 
very large portions of their budgets committed 
each year to continuation of existing activities^ 
which considerably limits budget flexibility. 
' o Ear-marking an4 Xiwe-t terns. While-many R&D 

agencies welcome positive Congressional concern 
for and support of their activities, it sometimes 
comes at the price of reduced budget flexibility, 
since Congress may earmark certain parts of the 

• budget to specific activities that it favors. If 

done on a regular basis, this may take the form of 

Congressional budgeting not of a single total sum 

• * 
to the agency, but of a series of specific sums 

• ("line itens") to subcategories within the total budget. 
For example, Congress regularly earmarks significant 



portions' of USDA research funds for research on 
cotton, tobacco, soy beans, and so on. . • 
o Formula-granting. Some R&D agencies, most .notably 
USDA and NIH, allocate a portion of their budgets 
according to a' flxed '|<^,rnnila. The USDA expends 
about a quarter of Its total R&D budget on a 
formula basis In support of the state agricultural 
experiment stations; NIH prpvldes certain Instl- • 
tutlonal grants on a formula basis* / 
o Intramural programB. Organizations with Intramural 
research programs find that the intramural budget^ - 
is hard to adjust; this is especially true^Aien the 
Intramural researcherjs have Civil Service tenuxre; 
Budgets tend to be determined primarily on the v 
' baals of the 'nu]d>er of intramural researchers and \ ^ 
their interests, rather than on the basis of ^ / 

changing needs, 
o Budget shrinkage. In times of budget stability or . 
shrinkage, existing programs tend tp take up most 
of the available budget » reducing the opportunities 
for new starts and redirection* 

Expand . Conversely, the factors operating to expand budget 
flexibility include: - 

o Finite program. The ten4ency for programs to 
develop excessive budgetary persistence can be 
counteraicted, in part, by establishing a clear s' 
time limit on programs when they are initiated « 
/To reinforce this approach, specific program 

goals and plans can be set and constantly reiterated. . 
o Single t%ne item budget. R&D agencies have an . \ 
obvious preference for receiving budgets as a isingle 
' sum, without Congressional or othei: specification , 
as to how it is tbbe allocated. NASA Is one agency 



that operates under thfe condition; consequently t ^ >^ 
the separate NASA Centers* receive funds to the extent 
that their pico grains respond to the agency's n^eds. 

a Diaox^H^ynary granting,^ As opposed to formula 
7 granting, discretionary granting retains the " ^ 
agtocy^s control over budg/et allocation among 
projects. / . 

o : Budget grcwtj;^^ Clearly, new directions oftd • i 
reallocations ate easler»'to carry out li^ times of 
budget, grovth, when new fund^ tiay be allbcatvr'.d 
freely. , . . V * \ • • 

HOW SHOULD PKOGRAM PLAMNING BE CONDUCTED? ^ 
During the pl^bmingr pTiose * of R&D mahaget&ent, two principal 
functions must be aCconplished: 

^- ■ • * ' ' ■ * '■■ ■ . ■ ■ . ■ ' " ■ ■ 

: '*^; o develx)ping nqio pi^ 

A defining the specific ob^jedtives and major paaka • * 
' of each new program ' 

. * - ,:,/;■, • ; ■ : ■ ; 

The manner in .which these functions are ^Vatf^ conaldr 
erably amonjg the various Bj&D ageiicles* those that emphasize the 
support of basic x^eseatch tend' to put little explicit and regular, 
effort into planning; their efforts are usnaliy ad hoc and responsive 
to the" concerns of the external R&D community. . However^ those having 
ah interest in practical and programmatic R&D> generally employ more 
elaborate and routine, planning , protedut&s / that rer^pond to the ^ 
interest of practitioners ^d other eventual rn^ers of R&D products. 

Hew yro^gram Ideas . 

Ideas for new programs come from a variety of sources , courted 
more Or lesis assiduously by>the several R&D. .agencies. In many 
Cases, internal staff; and mai^ia^er^ g^nerat^ most hew program ideas 
themselves; ip a hi^miber of case^^^'extemal practitioners and 
sciejitists play a lafrge role in* the establishment of new pfograms. . 



ititernal Sources . Ideaa for new program starts arise within 
l&p agenciiss from program inanagers, Intrauirai scientific staff > 
and the agency's^ administration, _ _ 

The progrcim managera of existing extramural programs derive ~ 
recommendations for new progi'a^^ in several different ^sys, including: 

o Identification of pi^mising redireotiona or spin- 

offs from^ exteting programs* \- , ■ . ^ 

Several /agencies identified this as the most 
* frequeift soutce of iiew program ideas. The Forest / 

Service, for examp^le, redirected on-going work on 
^ - disposal of pulp and paper waste in order to Study ^ 

^ the effects of distributing sludge 'from waste ^ 
treatment plants oa forest floors, when the latter 
idea was proposed. The National Institute of Child 
^ Health and .Human Development separated a program 
on i^opulation from an existing program of reprb- 
-^ductive studies wheii it l^ecame clear that- the- f 
: 'gove;rnment was becondng int^^^ " ' 

problems.:^ ■ '■ * . ' .- 

o Detection of dtcorging diredti^m 

^^^onmmity., . ■ : : ; > ^ : ^ 

In basic tesef rch agenci^ ; stidi ^?is NSEian* NIH, 
new program areaa frequently follow,"^ : 
lead, the research \ititerest of the >asic r^earch <^ ' 
communities^ When program- loanage^s det^ect. t^anges • ^ - 
in the activities of a field of science, they 
translate them into new a^tocy program areas, p 
o Perceptions of critvcaV needs am opportuniHes. ■ 
In applied £Mid programmatic research agencies » 
, program managers -.should at tenqpt to keeg abreast 
, of the need6 of th^ gxactltibaers and^-o^^ users ■ 
'and the oppottunitiep opened by new RiD develop- * - 

mentfl.^ That information should suggest new.o- 
programs. ; .V-'' . • ' 
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This and the preceding method rely tq;>on the 
.Individual perceptions and Judgments of program 
o^flc^rs. Some agencies atteoopt to proceed more 
formally) especially in the idehtification* of 
need. , * 

Analysis^ of data and forml studies. 
A variety of different kinds of studies can be 
tmdertaken. to determine needs that R&D can meet « 

For example^ NIM& is considering using biomtrlq 
data to gauge critical needs for resemrch. In 
one instance » observation of declining state 
mental hospital and incteasiiig community mental 
health center cas^ loads ) plus data showing that 
blacks are not using community centers » aas led 
to research focused on learning why blacks do 
not use the centfers. 

NIMH also conducts small surveys of the views 
concerning critical R&D needs held by attendees 
at research conferences ) panel loeetingS) advisory 
council sessions^ and so on.' , * 

NSF's Division of Research implications (RANN) \ 

has a group that sponsors problem assessments 

- \ ' .. . - 

and exploratoiy research intended to determine 
the ]need for and character of hew R&D programs V 
Intended to respond to major national needs. 

NSF- 8 Education Division has been concerned with 
developing new pre- college and underjgraduate 
curricula in the sciences. The identification 
of -new program a teas is assisted by maintenance 
o/ a matrix of sjJ^j eat matter versiis lenei of 
schooling. As curriculum materl^^ls^ teachet 
ti^aining and retrainings and diffusion actlvitieiS 
are completed in each entry .of the matrix, it is 
checked off. Prpgram funds can -then be targeted 
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: to the blank entries. Before programs are under- 
taken, howev^^r^ they are also impacted to satisfy 
^ . the requireznents that there t^e a need for the 

curriculum that is perceived by Congress, the 0MB » 
and the eventual user community; and that th^re be 
an available development team that is Vight for tue. 
Job. 

. Yet another, approach to identification of R&D 

program needs Is e-OalAAatian of exieting operational 
programs. ^Deficiencies Identified during such 
evaluations suggest R&D goals « 

. . • * " " " "• , *• 

In agencies that haye introfmral etaff reeearcliers who may 

become extramural program managers, «new prograin ideas qftien come 

from the intramural Staff.. 



tr^v 



At NASA-Goddard new program ideas can and often do 
arise Spontaneously from within the intramural 
staff. Since the staffs is kept aWare of NASA^s 
long-range plans and priorities and since Goddard 
has both managed ^d conducted prog^mmatic ; 
activities for over a decade » these itleas are 
\2sually^lose to the mark, the staff member can 
usually devote a few man-r^months of effort to 
developmei:\t of the idea without need for high- 
level approval. , 
At OEO's Plcuining 9 Research » and Evaluation Division » 
ideas for new extramu:i;al programs occasionally 
develop from the studies being conducted intramyrally 
by In-hduse staff. For example » a program to 
<2xperimeht with various^ forms of health care delivery 
for the poor grew out of an in-house review of the 
OEO income laaintenance experiment. 

At. NIH, intramural researchers may suggest activities 
to be* parried out by outsiders through NIH-'s 




dlaboratlve research mechanism (which Is separate 
:om extramural research), ^ These are usually 
ictlvltles needed as a part of intramural research, 
or deriving from It, for which Intramural skills 
arfe Inadequate or Inappropriate • At the National 
Cancer Institute these collaborative programs play 
a major role In carefully-'deslgned S&D programs 

directed toward specific jprqducts, 

. .. 

* • . ' * t ■ 

Every agency^ of course, finds Itself subject to some extent to 

^ the S&D Interests of its adnrCnistration ^ most par ticularl]^ts . 

directory Some directors are content to ^erclse their interest 

through selection among tht^ ideas brougiht forth by the staffs others 

often suggest new programs. In the case of OEO's P^&E, the directoi: 

and deputy director's suggestions have a special relevance and 

value; since each of them serves on high-level policy tastic forces; 

each brings back to OEO's policy-oriented reseatch program a strong 

feeling for policy needs and priorities. . c . 

External Sources , External sources of program ideas are 
more varied than internal Ones. Sonfe ideas come from , operating 
agencies of the government, some J^rom the broader practitioner 
and user communities, and some from the R&D community. ' 

Three R&D i^^encies that receive program ideas froja operating 

(Xgenoiea are the U.S. Ait Fofce, NASA, aii^NSF-RANN. , ' ; 

The Air Force has formal procedure^ for obtaining R&D program 
'-.«■ ■ • ■ , V • ' ' • 

recommendations from the operating. commands. Any^ general officer 

; * ■ ^ 

in any of the operating commands can begin, the process by stating 

a perceived need to the Director of Requirements in his command. 

The Director prepares a document describing the Required Operational 

.Capability (ROC) and'sends it to his counterparts in the other ^ 

commands for coordination, and then to his^ commanding olff leer for 

» . ■ ■ • ■ . ■ •- 

signature and transmittal to Air Forcev headquarters. At Headquarters 

'■ ' . • ■ * " , * » • ■ 

• it is assigned to an action ^^of fleer who sends It to the Deputy 
Chiefs pf\Staff and to subject area R&D panels (e.g.. Strategic, 
Tactical, Airlift) for preliminary comments r He then pr-e^ares. a 



position paper that is distributed with the ROC for concurrence . 
(or disagreeinent) • The position paper plxis responses are then sebt 
to an R&D panel for formal review.^ If the panel approves the R(.C» 
a series of Required Action Directives (RADs) is issued calling ^or . 
supplementary studies of various kinds. Management of the j^r^tam 
is turned over to a Program Elemtot Honitor vV o oversees execution 
,of the RADs. Once the RADs are ccTiiqsleted^ the proposal receives 
two final reviews by the Air Staff Board aad the Air Force Council. . 
If both are favorat>le» a filial RAD is issued to^ the Air force 
System Command (^he R&D agency) to develop the final program plan 
called the Concept Formulation Package/Technical Development Plan 
(CFP/TDP). ' V 

IfASA's procedures are considerably less formal » at least Inso*^ 
far as they affect the Goddard Laboratoi:y. Each year » all of th% 
NASA Center directors meet to produce a long-range pl^.^ They take 
into account both technical capabllitijBs and political and 'budgetary 
realities^ "This serves as a framework against yhich th^ Center . . 
directors consider new program proposals. In some case8» it may 
lead to new program initiatives. In many cases^ it serves to; ratify 
new programs suggested by the Centers^ ^ 

NSF'^s Directorate of Research Applications receivies program 
ideas from fed^al^ state^ and* local, operating agencies. The process 
is only partly formal. Each Program Director has an advisory . 
coimQlTttee of federal agency representatives ti\at reviews his program 
plans and suggests ai^as of R&D .need. V Most program directors also 
consult on an* ad hoc basis with officiltls in Other federal af^ncies. 
But the program directors also responif to the"^ program id^as chat 
arise within the R&D coimDunity, partly under the Influence of \ 
operating^ agencies. • 

, * Practitioners are an especially important source of progi^am ^ j. 
ideas in agriculture. Their influence is strong in large ineasure 
because there are individuals — • 'the county agents; ~ who serve 
as links to the R&D community, who. are in close and continuous 
contact with the faziners. .The county agents- take as their rQle ^ 
hot only the dissemination of new products ^^and processes to n^e / 



ftfraersy but also the transmission baok into the RU) systen.of 

Information about the fanner's probleios and needs. They convey 

that Information to extension specialists and research personnel ^t 

State Agricultural £x;;>eriment Stations* %itio in turn may suggest 

ne'^r or redirected research p£gtg^ams at their local stations, new 

regional programs conducted at all th6 stations in a region^ or 

special programs attacking major national p rob lems » ^ such as com 

blight. ^ ^ . 

Industrial growers » some' of whoro cmtribute •to the support of 

S&D ut the* agricultural ^e^eriment stations t have been quite out«^ 

. . ' ■' - " » 

3poken in demanding that the stations undertake programs that 

• '■ • ■ • * 

respond to their needs. For example » after the station in a' western 

state developed a? vegetable ideally suited ko canning, a large 

packing firm in the midwest^iSjBiejdanded that the locial station develop 

a similar plan{: adapted ta^%idwest conditions. 

The R&D eonmunity wakea ^ts ideas for programs known in a variety 
or. ways, both informal and formal, In most basic reseaifch agencies, 
for example, the scientific «community is the' primary source <.of 
program iKle as, which it . provides Implicitly through the cum[ulation 
of its unsolicited project proposals-. At NIH and NIMH, the advisory 
councils (comprising primarily research personnel) of the instltut;es 
make program .^recommendat ions. Frequently, they serve as conduit/^ 
for program ideas suggested by -scientific practitioner and public 
constituency groups, such as the American Cancer Society or'medical 
scliiool deans. (Advisory councils have been discijssed in greater 
detail earlier.) * - , * 

But many of the research agencies have found ttie heed for a 
somewhat more orderly developmrat of new program, ideas*, , while still 
recbgnizing- the preeminence of the scientist as fbrmul^tor of 
program and project concepts. The almost universal response to ^ 
th±s need has been the convening W some form of ad hoa group activity 
committee J panels, workehop, or conference^ in which scientists are 
the principal participants. > . * • 

^ The NIH employs these mechanisms quite frequently. NIH admin- 
istratloh with whom we spoke felt that Bmallj informal pcoielo were 



*1>y far the best'^ of the possibilities for assessing past and present 
rese^ch in an area and siifcgestlng future research* Conferences 
have been tried, but tleir group dynamics often ^ turn out tb be bad* 
Individual consultants, while helpful « cannot cover. 1^11 the program 
areas at once or challenge each other*s ideas face-^o'--face. Thus, 
it is better to bring them together in snail panels, confronting 
each other informally. " 

NIH*s experience provides some guidelines for organising 
such panels: * 

o form a* small committee (3 or 4^ of consultants to 

pick the panel; 
o select 10-12 men in the area to be assessed; tbey 
• should be the-"creain of the field"; ' 
' o ^ tell them that their Job is to assess research on 

the topic in th^ past and ..the present, but laost , 

importantly, to say where the next research should 

be;' 

o have a top-^flight external scientist chair the 

laeeting, but be ;sure the program director prods > ^ . 

the panel to ke^p it going in the ri^t directions; * 
6 have a stenograj>hic record kept^ then have the 

« program officer distill panel commients and seitQ the&»^ 

to the panelists for review; 
o use the revised proceedings internally in program 

planning; ^ 
o/ if stimulation of ^^^jjj^^^ and proposals is desii;e4» 

have a version of the panel's assessment published 

in a professional journals 

, Definition of Program Objectives and Tasks 

Onc^ the idea for a new progradk has been accepted, its specific 
objectives and tasks must be laid out in sufficient detail to perxnit 
project activities to be undertaken and supported* R&D agencies use 
a variety of methods for elaborating program ideas » which we have 
^categorized as ail hoc, semi-formal, and formal procedures*' 



Ad Hoc > In agencies supporting basic research » ad hoc program 
elaboration nay coioprlse nothing siore than ah allocatrbn of funds to 
the pxxigram area to await the unsolicited a^r^als of project proposafl^s 
from the rescarch^'^cbamunlty. Some agencies take delicate steps 
toward solicltlng^uch •\insollcl ted" proposals *by: ' ■ ^ - 

0 announcing their areas ^of interest with some 
\ ^ specificity » 

o holding cohferences whose prixoary purpose is to 
generate ptoposais in an area,* 

o directly inviting particular researchers to submit 

proposfltla (this runs some risk of enbarassment^^hen ^ ^ 
project proposals are reviewed by an autonomous review ' 
panel and, as, sometimes happens, they reject the 
invited project) * ' ^ . 

Hie more structured ad hoc pn>cedures generally Giq>loy an 
external panel or aa intern;^; task force, wtiose meiobers have been * 
selected. for the*purpose> to suggest *the program's necessary projects 
and objectlvee. The external panels are generally similar to. If / 
not the same as, the Informal panels described earlier as 1the source 
of program ideas. * ' , 

The result of these ad hoc protedures Is generally no more than 
a collecti^ of desirable projects aud, perhaps, an indication of 
potential jperformers. "^e plans rarely, if ever Interrelate the 
CGiiq[>on6nt projects in sequence, indicate critical subobjerctlves, or 
estimate costs or. time. 

' Semi'-foy^l . *ftie semi-for^^l procedures used for program 
elaboration as a rul^, at^tempt to describe in greater detail \ 
what each project is to do, for how long« and at what cost* ' Ih 
contrast to the formal procedures to be described bolow, however, 
thfey rely on subjective and unstructured methods. " ; ^ 

When OEO's PR&E^dlvlsion has ^decided to undertake a particular 
jflannetl evaluation ipif a social program, for exasq>Iev It lnltiat«?fi 
c-ycareful. i«i -house proliiilnapr planning, to detorthinc: • * /\ 
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o Vhat to find out? (The eaphasis 1$ on separating 

the esaentlal^from the dealrable«) 
o Sow to find It out?" ' . * 
o How diuch- it will cost? (Costa are estlinated to 

, within 501.) ► 
o Who will use the results and how? ' 

A two--pcrfton tor^'^j tonBlstinQ of an an,Tilyst and a pi^rsori knowledgeable 

about the prograif area to be evaluated , carries out this plannlns, 

.employing consultants as needed. At the end of 2 to 4 months » a 

J" ■ ♦ ■ * ' 

Request for Proposal i^iFP) Is written that wll^l be used to solicit 

propt>8als from prospective contractors for the detailed design and 

Inplencntation phases, of the evaXuatlot), 

At Nltflf when a new progress area responding to a critical n^ed 

has"been ^identified » an effort ±tt made to detennine whether the 

laformation needed is alttsady available^ throu^ prior researdi* 

S^teioatic search eUforts are undertaken. Each program chief is ' 

re6p<»nsiblc for: * 

o screening the relevant literature in his's^a/ 

o preparing suanaries of Innovative information ' 

litficovcrf^d and project^ visited, 
o screening all NIMH projects already funded* 



If the needed information is not all ayaileble; then Investigators 
will be "stimulatet^^Vto apply for* a grant. . * . * H 

The most orderly of the semi«*^ormal pt1>c^dufes 1^ th^^t employed 
by KASA^^Goddard. There, any meafcer of the staff can initiate" a 
program idea, which then^^t^rogresses jthtou^ four distinct* phases 
from conccptlin <o hardware* At the end fif^each of the first three 
phases; projgrefis^-ls revised and^the decision to cohtinue or * 
terminate/ lirjmr4e. Th« phases are: 



o A Idea ^tage 

.0 E dfcvi^i«^pfe<;nt alternative! ^pproaphesi • . ^.J 
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o C — detailed project design 

o ' D — hardware fabrication and operation 

Phase A activity* 'averages 3 to 6 mdnths durdtlon» is conducted 
int^|iially» and produces/ a concise statexoent of mission objectives^ 
idj^tiflcation 'of major research and te^nical requirements, assess- 
ts of feasibility and desirability » and sketc^ies of alternative 
proaches to fulfilling mission requirements. 
/ Only ^out 10! Phase 9 planning efforts are, approved by the . 
y 'G^ddard director eaah year. \ By this point, the required effort 
becomes "notlce4^1e^' to f^i^CopgressV* and the like, so NASA . 
Headquarters receives copies of edch Phase B plan and the project 
entcris the "new start" category, • Frpm 6 .to 12 months and from 
10 1^0 SO persons pre required to produce, a report on alternative 
ways oif achieving'the mission, identification of state-of-the-art 
constraints, abd estimates of development time and total cost. 
This is/ generally done ii;itemally, although in so^e cases to 
half may be contracted out. . - , • 

' Formal . The fo^nidl procedypes used for program elaboration 
generally attempt to i^^npose a regular structure upon the process 
of decidijig what projects must be carried out to achieve a program^s 
'objectives. This structui^ seeks to Abreak large subjective judgments 
into smaller ohes that can then be combined in ^oine obje'ctlve 
mannet^; tlic restitlt is usually dl^Xayed in d highly specific format. 
The acVantag^ws of such procedures is the discjj)llne that is impoised 
on th^ p3,ianing process; the xfesult can be Impxpyl^d interproject 
coiopa^lsons. The disadvantage of these procedures is the pro- 
.QTustean nature of the bed"^ into which program pliahnlng is foxced; 
^more may be lt>st Ija^jexaludihg implicit judgments than is gained In 
explicit evaluation. • > \ ^' » - 

. ^ The National Cancer Institute J(NCI) has developed a formal » 
planning procedure, the Cfinvergenoe Technique f\thBt provides the 
framework for planning ■collabbraCive R&D programs intended to 
acfjieve specified goals« Its product Is .a oonvergenee plarif displayed 
in a cak7Ve2'g0nae ahax't;. The ch«irt comprises^ three separate arrays: ; ' 



o the lineca* arvay comprises the.ialnlm\m sequence 
of research activities needed to achieve the 
p*rograin'8 abjective V 

o the concurrent array comprises related, but not- 
essential, research activities that ©ay yield 
itiportant insights 

o the supplemsntary array includes **blue sky" 
projects that may have big impacts, lut with 
low probability^ » • 

The design of a convergence plan is a 4 6 we^k process,, A plan- 
ning group I containing 5 to 7 persons can c^rry it out, 

^ The first ^task is to define 'an operational objective whose 
attainment would clearly end the 'program, \ . ^ 



^o "Cure leukemia" i-s npt gpoS. • 
o ^'Develop vaccine for acute leulCemia in early 
childhood" is oi/eratlonal. 



The second task is to defifle several intermediate goals between 

the oltjective and' the current state*. The intermediate goals define 

the program phases. * . • : • 

. . -I' ' ■ ' ^ • ■• ■ . ■ 

the third task is to detepnine what oxn^teria would have to be 

satisfied/ to justify the decision that an in^'ermedlate goal had 

been ^teap^d. / . . ' ' ^ 

The fourth" task is to derive fi?6m the . Qriteri^ the seH^s of 

pToJecte needed to pt-tfvitfe the InYormation tp 'satisfy • tbjfe "c^'iteirist. 

Tlie resultant convergence plan^ (and .chart) is* iised to manage 

the program -as it proce^d^. New, projects related ^ta a; program can 



gene^lly be entered in .vacant; sedtions in the chart und^r pre-- 
defined headings. « 

/ A .number of industrial firms have^deveiogisd and employed 

' / ' • ... * • . • , . 1 •* 

formal, R&D planning procedures that we shall refer fto, gerierically , * 
cs; scoring models. ^ . j " .r 



These models vary In theif' cor^lexlkty , although each aims to 
serve the saioe purpose: to select from amb^g a set of proposed R&D 
acti d-ties those that maxiMz^ sonie ,ineasure (to the corpora-^ 

tlon li\ the .indust rial application) . ^ ■ ; i ^ * \ ' 

The fii;$t step iiiF each of them is to ti^i -i/ife^'f^^ 
to be carried out as part of the program. Ther^^l<^)nd s is to 
employ expert judgment (of staff, consultants, or^>^utside j^anelL ^.o 
•estimate the a cores of each project on a number of cajt^xiaj iu^ 
time-phased project costs. Among the criteria might bef^^^'^^t^^^^^'^^^;^^^^^^^^ 
■ * ■ / . ■ y ■' " ' . . . ' ' ' ■ \ ""V-::. • •;■ • ^ 



o benefit in dollars of project completion 
Q relevance to program objectives 
o ' probability of success 




These scores are then cofflbine,d into an overall s,<^ore for each project. 

The third ^tep is to select those projectrs that maximize some function 

of the sioxes subject to budgetary and time constraints. (Mathematical 

proccOares would be used to. do this.) 

Y One comp^lex variant of these methods seetes to link R&D project^ 

with the attainment of a raxiked list of operational objectives. 

Each project is 'described in terms of its criticality in serving 

edch objective, and itD costs. A computer program^'tKen determines 

.how funds shbultf be allocated among tjie various projects to achieve 

•. ' ' ■ ■ ■ ^ , _ ^ . 

maximum attainment of the operational* objectives. 

HOW SHOUp PROGRAtt DEVELOPMENT BE CONDUCTED? ' . . . 

■ ' During the program developmnt phase of R&D management, two 

■■*•*.* •. .... 
i tasks" must; be accomplished: . .-^ i'" ' * 



o q^eleol^ion of projects to be supported. , ^ . - 
p. monitoring ^ the progress of projects that are . . « 
' . underyay / ' * ' a ' ^ 

^Project Sel€fcti3n * . ' ! • * 

. ,the task of selection :vampng project proposals differs according 
to, whether a grant or' a ,oontraot is to be awarded, since the. traditions 



and legal assumptions associated with these two forms of support are 
quite different, » * . 

The grant has ordinarily been the method used to fund basic research 
proposed by the' researchers. Selection among project proposals has 
generally been on the basis of a judgment of the/ scientific quality 
of the proposed research and the researcher. 

The contract has generiilly been ^he.method used to fund programrr : 

matic R&D responsive to the. needs of the agency or its constituency. ' 

. '■■'* 

Selection among proposals has had to consl<^er both scientific quality 
and prog^m felevancp.^ 'as well 9s cost, different selection procedures 
have evolved to handle these different situations. 

Grants . TKere are four tasic piocedures for selecting pro ject^. 
to b>e fliaded by means of grants. .These all rely upon a program 
officer (or executive secretary) plus: * 
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o ' no outside reviewers (tha*- is, the decisitfn is 
the program officer* s alone) ^ V 
o maiZ i^evieu) by outsiders 
a ' a sin^ie ponei of o'^utcide reviewers 
o cS^al poni? is of .outside reviewei;s (operating one 

' after the other) ^ . ^ 

' ■ ■ ^ .0 • ' . ' ■ ... 

The, no, outside x*eview4r ptoceinre is employed primarily at the / 
__ice of Naval Research (ONR).; It depends^ upon having program. 
ofHters who are in close and' continuing contact with the reseatch 

■ ■ " ■ ■ 

ai>d researchers In their filslds of Interests and who remain abreast 

of the program concerns of * their agency. The prbgtam pff icef IrVijlies 

upon his* gvrh judgment .and his general knowledge of the state-of-the- 

art and the Wputation of individual researchers to docidit whether 
• • ' ■ "^^^ ' " ■ ■ . . ' • ■ ■ , * * . 

oip not to support a proposal. To be effective, consequently, program 

officers must spend ^considerable tdfne keeping abreast, which means . 

traveldng and reading. This reduces the number of proposals each 

cap handle. 



The advaniages of this procedure are: ^ 

. •••• . • ■ ' ^ ■ /. • - • 

o the program officer^ is 'free to develop a program that 
responds to hio perceptions of agency needs; ^ *\. 

o the program officer can bet on newcomers, mavericWb, 
and outsiders who might be* excludeid by reviewers drawn 
from the most prominent in an existing, disclpllti^f ' - 

o ' the program can: change ,quipkly , in response to new n^eas 
or opportunities; » 

o the program officer's judgment of quality can be moxe 
• - rigorous than that of reviewers drawn from a weak ' 

jiresearch area. (This can be of Considerable importance 
in sbm&^areas ot educational R&D and the applied be- ' 
havloral and social sciences In genetal.) : 

6 the progr^un office; can codbine an awarenes's of specific 

^ practical problems requiring R&i) with his jugemeht of 
RScD quality In a way that few researchers can* 

The disadvantofjes of this procedure are: 

■ • ^ •■ \ • . ' - ' ■ ■ ' ■ • 

o its complete reliance upon the judgments of a single 

person, whose personal strengths, weaknesses, and 

prejudices affect an entire, area of R&D; 

o ^ its vulnerability to favoritism and improper j.udg- 
ments and, therefore, to criticism from ^ientlsts 
and from Congresa; - , 

o its low profile In the R&D community does not gen- 
^rater>the natural, ponstituency support provided by 
scientisits w.ho particip|te in the panels employed 
by other "procedures.. , 

The mccil review' procedure is exnployeddn the physical, math- 
ematical and engineering sciences branches of the Na*tlonal Science 
Foundation, with tVie exception of the bivOlogicai sciences.* The • 
cognizant' program officer sends each proposial he receives to at 
least 3 or 4 reviewers, chosen ad hoc to be appropriate to the / 
topic of the proposal. A complex proposal may be sent to several 



different groups of revi::wers for comments on Its several aspects. // 

Reviewers do not generally ^ee -Inpre-.^than one proposal at a' time; 

thu^l^'ihe reviewers da not provide a comparative ranking, according - 

to th^^same criteria, of ^IJ, proposals arriving at a cettain tlm^^^^^ 

Th^ prpgralh officer makes the comparison among projects as. he; drawd|^ 

upon- the reviews in deoiding which p^rojjects to fund. Suimnaries of >/ 

■ ° ■ ' . ■ . V, * ' ' ' 
the reviews, together with his decisions, are then'^fdrwarded by ^ 

it * ^ • « ■ ' ^ ' ■■ 

he program officer to his adrolaistrative superiors for a. rovai..^ 
Two variants of mail xeview have been employed at NSF. 

o Delphi vartant. One variant, is an adapatioh of the , v 
"Delphi Technique," which employs an Iterative series of > 
questionnaires and feedback about prfbr 'quest ionnai^sp 
results tsy produce a better informed group judgnfent , • 
^ from a group of experts. , The mail review/adaptatlon , 
is, whfen first round reviews disagree, to^end to.each 
-^reviewer anonymous syjnmaries of all the reviews "and 

theq hold a second round in which each reviewer may ^ 

' ' ■ • ' - *" • . . ■ 

revise his review in light of his colleague's 

■ •*■■■#' <i • ■ , • 

;comments«. • * , . . » 

. . ■' ■ - : ' ' i . • 

d : Ppst^oj^opd iaudit. Another variant is to have a 

program advisory panel meet eviery six months or so to ^ 
go <)veas'and comment 'upon the actions taken- by the 
program officer', who can^bfe given a little more flexibility 
i to act quickly, and make 0ecision*s on his own, since 
*^ he knows hie actions will be reviewed subsequently. 

The adyantagrea of mail r^vjieif are:, . 

. \ . • . _ . . . ■ . • . 

p the agency can obtain the adyicer. of scientists' who 

could ])ot be attracted to ^permanent goveriiment * 
service; ' 
o ^; the choice of outside reyiewers (including 

. practitioners) can be tailored to :the .topic of each . ' 

^ . proposal; , ,/ . 

o Jevery reviewer- has his say, Vithbut the chance df^ .' • 
being dominated by colleagues with strong' ^ . 



personalities or large reputations as sometimes. 

• *• . ' . . • 

^ occurs In panel meetings; ' 
o the work load of review can be spread over a 

great many individuals, instead of being concentrated 
on.^ few panel members; 
p « decisions are not subject to the prej indices and 

•narrow viewpoints of a fixed panel; 
o decisions are not as Vulnerad>le to a program officer *s 
weaknesses as they , are in the '^uo outside review" 
jcase. -■ ' , \* . • 

The disadvantag^a of mall' review are: . '^"^ > 



o th6 difficulty in obtaining and maintaining lists of 

appropriate mall tevlewers; 
o :the lack of interaction among reviewers *.(exc^ept in 
^^:^h^ Delphi variant) thtough which they can' work out 
disagreements about a project's quality; 
tlie ease with which reviews can' be orchestrlited by' 
a parogram' off iceic through judicious selecti )n of' 
reviewers (some wpuld count this as ,an advantage); 
0 Hihe lack of opportunity for the reviewers to c6mpare 
anil rank groups, of proposalS|^^*.whlch tnean that 
differient proposals will be evaluated according to / 
different criteria 2^d standards of judgment; vV 
3 the labk of direct cotitact betweefi^j^rogram ofllcer 
' and reviewer means that the reviewer may not 

appreciate the program's objectl;yres and priorities* « 
or know, aliput its other '^cflvlties; mo repver, the 
mall reviewer is less llke],y td become a>^strorig 
program advbcate than is a panel m^tjiberi 
0 many mall: reviewers tend to.si^e /a proposal* only in 
terms of . their own fi£l£;./jsome"" are mostly negative, 
because they downgrade every^de else's w6rk; some 



are mostly .positive?, because they feel theli: field 
is Important, an.d should be. expanded. 
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The single panel pfocedure is employed in the biological science 
and social s,cienc€fs sections of NSFy-^iend in other agencies on an hoc 
basis. The cogi^lzant program officer presents a group of proposals 
to a panel compirising*' 10-15 scientists. The-' proposals mj^y be only a ' 
selection of*tho6fe receiv^^d, generally the most difficult to decide. 
'In the NSP system the pa.als are primarily .advisoxly; they do not 
formally rank or approve project^. The program officer, however, is 
guided by the judgments of the panel. It would be possible, ^cf course, . 
to have the panels score the proposals and then make awards. in order 
of scire, but tuat is hot commonly done« Th^ program officer decides what 
projects will be funded and then forwards his decision to his supe 'ors. , 

Therfe a^ : two forms o'f single panel review. - . ^ 



o Fixed panels* In this fotm the panel is regularly 
constituted in a Subject area. (om> panel per program 
- ' ""officer) and meets a few times a year to' review 
proposals. Panel menibers may serve 2 or 3 year 
t«ms.- 

o A\jl hoc panels'. In this form, the panel is specially 
set up to deal with a group of related proposals on'' 
a one"-tiroe basis. This may be done when a new area 
is V&ih^started, when enthusiasm for a particular 
tdpic p^aks rapidly, ox when a subject requires a 
special,, mix of specialists to give it proper review. 
, . * . '■ ■ ' 

The advantages of single panel review are; 

o the panel comprises, many scientists who could not 
be attracted to full-time government service and ^ 
program managemen^t; c 
o the panel has access to thejfull range of proposals 
and judges* them spmparatively according to the same* 
criteria and standards.; * . ^ 

o 'the panel can ask for and obtain additional* 

information about the program or proposals^f rom 
^ the program officer and each other (some* may have 



.^personal knowledge of the proposer); 
o the program officer will benefit directly from his 

contacts with the participants and the meeting; 
• o the panellsts>are more likely than mall reviewers « 

to become advocates and expositors of the program 

among their colleagues; * . ^ 
o the panelists have an opportunity to draw upon and 

dispute -each other^s judgments; 
(T the panel (If It cons^ris^s prominent scientists) 

provides assi^ance to the scientific community and 

Congress that decisions are being made fairly and 

pbj'ectlvely;.. ^ ^\ 

o decl^d^dns are not so vulnerable to program officer's . 

Meeknesses as iri the 'V;o outside review" procedure. 

; * \ ; . ^ ■ - 

The disadvantages of single panel review are: 

o the pajiel may be: dominated by scl<£ntlsts with strong 
persoilalltles or large reputations; 

o panelists, since they are generally those who have 
succeeded under the existing resei^rch approach^ may^ 
downgrade^ research that explores new approaches,^ , ' 
combines several dlsciplihesi or in other ways is/ 
outside the norm; 

o if the research community in a subject area is we^k/ 
^ , ir may not be possible to obtain high enough 
' standards of judgment from a peer(^anel; 

o If the panel's area of responsibility is broad, it 
m^y not have^noagh members who can comment on a 
specific proposal to.give^a group evaluatlorf; ' the 
results is that one or two panel members sp^^ak for. 
the panel in that area, yet express only their own 
preferences ^nd priorities. 



. .. The dual panel procedure is employed throughout the National 
Institutes of Health and (with slight variation) iii many parts of th 
National Institute of Mental Health. 



Prdposals arriving at;NIH are sorted, by discipline to one of 4*8 
etud^ sections^ which comprise groups of prominent scientists in each 
discipline. Each study section has ^avehatjMa?!, who is* an outside • 
.scientist , an executive seoretdrg , Vl^o* is the NIH program officer 
for . the study section/ Ijhe study sections reyxew and assign' a numerical 
score ("rank") to ei^fch proposal. The ranked proposals from the study * 
sections are -regrouped, and the scopes rescaied so' as to achieve compara- 
bility, according to the program interests of tlie 10 NIH Institutes and 
forwarded to the appropriate institute advisory oouncil^ which comprises 
scientists and practitioners. Each institute has several pro-am ' ' 
direotovs i each of whom is responsible for phe proposals in one subarea 
of the institute's* concern. The program officers present . the ranked ^ • 
proposals from the study ^ection to. the councils "for approval. The 
study sections are s.upposed to^ judge proposals solely 6n the basi^ of 
their scientific quality, while the advisory courfcila* are supposed to 
review their reliance to the institute's program, IhSpractice, most 
proposals, are paid in an order determined by their stifdy section ranking. 
The exceptions are projects jjudged ta have high program re^levance (HPJO 
-despite a relatively low ranking by the study secti^on. They may -be , 
paid out of order by the council. * ^/ ^ 

• NIH staff (the ex€:cutive secretaries . an c} program directors) play 
a much smaller role in thid selection process than of their opposite 
numb-^rs at l^SF or ONR. Most of the decision-majcing power, is in the 

hands of the study- sections, and theoreti;lcally* at least, in ^6 / 

^ ' ' ■• • '\ ' . ■ 

institute councils. Executive secret)£iries organize and manage the 

review process, write up summaries, and recommend new study section 

mesnbers; program directors "solicit" ne\y work, and handie the technical 

aspects of the institute's contacts with existing projects to be funded. 

•* • . . " ■ ■ . ■ *■<»/' • ' 

The advantages of - dual panel review are Jfettose. of sin^Ze panel 
review p^lus: < 

o * an even gteaner reduction "HHf the chances for irqjroper 
program officer influence since the stimulation of 
uew proposals (by the institute program directors) is 
ppmpletely insulated from proposal review (by the 
study sections); . * . ♦ 



o a prospect of greater efficiency because of the 

greater specialization of function among study sections ^ 
^ institute councils » executive secreta];le8y and prograrti 
directors; . , " 

o an. enhanced prospect of ^ objectivity in each panel's 
decisions because the other panel is always looking 
* ^ ov^r its shoulder; 

^o a potential advantage over single panel review 
I j[vbl(^' generally esiphasizes scientific quality) in ' * 

the prdvision for explicit judgments concerning . 
^ program fe^'eyance by a group specially constituted 

' ; ^ -r four, that .l>u]n?o8e; 

6 in enhanced^'assuraivce to the api^^ants that their 
proposal's wi|l be judged by competent scientists on 
. tcienti^fic 'grpunds; v 
' o ' aii 'et^ani:e/i feeling of participation by the , 

' scl^tists that incre^es their sense pf responsibility 

fbr the* aigency *8 programs and their activity in building 
^ the agenc;^'&; s cient4.fi c constituency; , 
o ^ an enhanced ability t9 support politically sensitive * 

projtCwS by reference to approval by a prestigious 
* * advisoiy council. , ' ' , , , 

The disadvantages of dual^anel.review include those of slngl 

panel revir-r vlth u^ia adidition that: 

study sections t^nd to undervalue activities in Vhich 

■■ . ' ■ ■ / * 
problem relevange is hif^i but^ prior research activity ' 

±s small or nonexistent » yet councils are cautious ... 

about' supporting proposals that hdve been ranke^d low 

by^ study sections (this lea^s to difficulties in the 

soc^ial and behavioral scictice^i and indicates that 

, ■ ^ • • • o ' ••• • 

dual review- may be more satisfactory in« the 

♦ * biological 'and physical sciences) ; 

o dual panel review leaves little control over program 

to- the agency^s internal staff, they become captive 

of their''Study sections.' and councils and frequently 



cannot move the' progr'iia in directions they feel are 
' . iipportant. 

In soae single or dual/panel review procedut^s, additional sources 
cf liAormtl on, .beyond written proposals are used in making project 
selections. The most lioportant of these sources Is site visiting* 

Site visits usually require 2-6 p^ersoas {lO. or. 12 in conipl^x 

cases) to spend 2-3 days traveling and visiting the principal investigatpr 

and his project tea^r, at their institutioa* Since f-hJs is expensive and 

tiiae-consuming, i^ is only done when certain conditions are satisfied , 

although most program of fleers feel that site visits save more money 

than they cost. For example, one agency site visits ouly for proposals 

tha^ are over $200 thousand, have significant luhbigulties; or whose 

deficiencies Seem to be due to poor writing. An&thar visits all 

« ^. . 

complex" proposals. Size, cosq^lexity , and the importance of staff 

interrelationships and institutional support wou.^d seem to be factors 

that shv^uld justify site'visits. r ^ * 

Site visits are tlifferent from. panel reviews because: - 

, o the permit the project to be examined' id far "^greater 
depth; 

o they reduce the importd*rce of di8C^>plindry biases, . 
because the project team presents all its research 
<in *^very area) to the full review team and there iff* 
therefore, less opportunity for biased attention; 

o they provJ.de an opportunity ^or two-way communication 
between the project team and the review team. 



ixtent 



The advantages are counterbalanced, to some extent by the site^ 
visit's greater qost in dollars and time (it may add several month§ 0^ 
to the time requireJrfor re^view) o 

Some agencies award some of their R&D grants on a formla-basia, 
rather' than on a discretionary basis* / " 

The most striking example of this 'f>ractice is the ljl^DA^s support 
of 53 State Agr^ .ultural" Experiment Stations* (SAES) through. Congressionally 
mandated formula grants (determined oy rural popul ntionV^ vhlch must 
be matched by the states and counties. This money j^^^vailablf^ to the 



r 



SAES to sp^nd ai ;they wl&h» subject to ooaitoring to see that the money . 
,l8 expended according to legislative Intent.' * 

The NIK also awards General Research Support Grarfts (GRSGs) on a 
formula basis to stable iiistitutione that receive xoore than $40 thousand 

discretionary support froK NIH. The formula provided for a GRSG t& 
be a percentage of NIH research grants at the institution; the percentage 
decreases as the total increases. The philosophy behind these formula 
grants ,is th^t nc^^ ini^titution has a perfectly balanced mix of research 
grants and that flexible s».3pport is needed to permit the dean to: 

p compensate for Imbalances caused by .the presence of* 
, eminent professors who get, more than th^ir share of 
research support or whose projects dominate a ' 
department; 

support young investigators until they build the 
credentials needed to obtain project i^suppdrt; ^ 
o support senior investigators Who >wi8h to fol^ow^ a 

new path; ^ - * 

o fill gap areas in %hkt Institution's research; 
o purchase new equipment; , * ' , ' 
o support research training prdgrams. * ^ 

r ' \ - * /. . 

The effect of this program is to increase che^power oi the dean*s 
office iu institutions where the project grant system has otherwise 
tended to increase the autonomy of individual professors* The' grants 
are given to t^je highest ^dministt^ative officer of the research ^ ' 
institution 9 schools that have sevet^ti equal ranking officers » ' \ ' 
receive the money 'in ipuiti pie grants^ oiie "for each fif ficer. Mosjt , 
institutioivs receiving such grants make active usi of them; in soM x.^ 
instancy the fund^ turn <nrer A or 5 times duripj^the ye^^ as tliljy ^ 
are loane^S4;a^^co^^ pfr^^ct costs^ until other jso^ : -es become ^yailablif $ 

Xh^ (iiJ^fiouitiesJUth thes<^ ^ants are th^tt the dean can use them 
fdr em^lre-buiidlng jrather frfiaii research; they <5an be jir^ed to siWtalh * 
mediocre, faculty Cho ai^^un^bl^ obtain jtroject siippdrC Cai^'^^H^ * 
many •institutions fmpes^^ tim^limits w' support frcrtt^ these fWds)';\^y 
tend to be useil for lower qualit^-and mox-e ri^y .projects tfeiirixroject -^^^ "^^^ 
grants; and ^ they are roor^ easily .cut andviaore Vlilnd table to critlcaK . 



0MB or Congressional scrutiny than project funds. > ^ 

Soiae agendles have found it advisable to ^t^llsh separate fttaot 
review mechanisms i^or amalt projaata (sayv less than $5,000)\ for Jorg^ ^ 
pro^ccte (say» mpr^^than $500,000), and for inter^^&criplinat^ pjp^ects * 
the separate mechanisms havei seemed desirable .because of the dif terent . 
«electi6n criteria applicable to projects iji these areas as bpposed to 
the usujil $50^000 grants in a single discipline. 

NIMH's small grants program makes awards of less than $5000 a 
yeaif;. ^TT\e gr.intn arc used to support pilot studies, the development 
and field :t<istlng of Instniaentation, and young investigators (who 
thl:uk they 'will not get an eqyal hearing in regular study sections) • 
One panel and a staff of 2 handle 500-600. proposals a ye^r. They v 
respjpnd in 2-4 months^^^as opposed^ months for dual review. 

» NIHH also has separate reviews for large, program projects -because . 
it found rigul'ar panels getting too bogged down in details of large 
projects/^ It seems that reviewers who are familiar with regular projects 
are unvllling to modify their criteria for the larger, more generalised 
^projectfi*. ^ ' 

The review of Interdisclplinjary projects causes special difficulties 



NIll £^tabli;$he8 separate study sections to make such reviews. The 
difficulty is findiivg proper reviewers, who will not be focused solely 
on t^jeir tilftclplinary interests*. NSF's RANN has found it« useful to 
.employ -representatives of the user and practitioner cQmmunlt;ies (federal^ 
State, and loc^ t>f ficlals) in reviWing Interdisciplinary problem** 

' oriented Research.' , 

• , ■ ■ V ». 1. . . - ■ * 

Contracts/ There are "a 4:iuai>tet"of different methods of selecting • 
appliedT^ai^l^programmatiq projects to be funded by means of nontrdata. 
ThreeJ models that span the range of variation are the ones used *f or: 

• ■ . • ■ "' 

o NIH ootlcborative Tesearch, ^ 

o^ NCI paf^jgrscmwatte if^cQeqrch, S - ; 

' o* GEO evaluattom. v 

NIH coVlaborative research was created, as. Its name suggests, to 
pe^nplt intramural ttese archers to collaboVato with extramural researchers 
on basic i^earch projects* Grants could have been used to support 



i 




; the extradural researcher » but cxp^ricncef sl)aved that under the ireedota 
providecf by the grant form of ^t^pport, the extramural researcher tended'^ 
to drift off oncu intet^stB pf>is own. The contract form of support * 
WHS, thercjj;^re, adopted to permit clo&er control of the research \ 
performed. ; Its use has ;^xt ended to cover purchase of research or \/ 
reseWth. support activities .that cannot be dQP^ sb well within the \' 
Instituti^^ such as larger testing, survey, or data gathering activities. 

NIH has the authority to bypass the normal competitive bidding \j 
procedure for awarding contracts » consequently *i't has developed Iti 
own procedures. \ . 

iSi^itute d^loys a dual panel procedure that reaeiD(>l!eB dual 
panel grant*' revi^^ u^ccept that both panels are creatures the 
Insltltute. • ; / - 

• ■- ■■■ ■ ■ " -■ / 

. o Hrat level review is provided by an ad hpc cqmmittee of 20, 
f * chosen to cover all' scientific .dimensions in the in&tltuHefs 
program.^ Subgroups *of the 20 are ce^cted to perform 
certain review t4$ks. . Three mendbeza perform an in^^depth v^, 
review of each l[>ropo8al« The co]p4ittee Is instructed 
to Judge 'contrac.t proposals on/^oth scientific merit ard 
" pxogram relevance. ^ To assur^ that the committee is 

productive, the institute director chairs in*, If the 
^ ^ committee suggests a sltio^ visit, that is aiftanged to 



take place; before the s^ond level areview committee^ 
^ meeting. . ' / ^ " ^ 

o Se&>nd levet x^vie^ /is provided a ^committee of the 6 
members of the in^itute dirfsctorate and 5 Senior - 
outside consultajdts (these are often Ibdividuals whose 
work on firfl(t level revidi committe'es was well regarded). 
The institute/director is chairman. They examine summaries 
. of the fi^s^ level <re views,' but generally only makj^ 
marginal adjtistments. 'Contracts are awarded according '^ 
to their decisions . ^^^The-^jrincipal advantage .of this 
dual pane^l- review; over the simpli&f methods: Is the « 
, arrangement of checks and balances^ that it provides. 



4 t 




The Mkttonal/tancer .Institute has developed an elaborate pVotedure 
for planning t^elect^ng^ and isonitoring collaborative research the - 
oom)ergenoe technique ^ Mhose planning procedtitcs have already been 
discussed. As parC of that process » rt^quests for proposals (KFFs) nay 
be issued to obtain cofintr act or ^assistance in specified areas. The 
responses to the RFP ar^ evaluated by a dual panel syd\en wlt^ 

apprc^riate program^ area. Fof^ example, within the Carcinogenesia ' 

• . ■■ -i 

program, the proposaul goes; ^ 

* '''■'.*■• 

^ 0 iitst,- to the Carcinogenesis Contact :jRcivlev Cpnmittfee, 
vhich is made isp of senior program i&anagerS) fof a 
\irevlew of relevance » needy a^ (the ' 

orders in which projects iit^ performed) ; 7 , 
o theh» to the Technical and Science Eiccellence'Xeview" • v 
Gommittfeee» which is ma(de up of outside cionsultahts» ^ 
(and perhaps chiefs o£ intramural research htanches)^^ 
/ * for a review of technical compet^feuce^ 

■■y> If the second review group jcannot narrow the choice^ €o;.a' single 
proposal, "diey select an od team to inake site visits ^d dj^dse- 
among proposals. • > * 




The Office of Economic Opportunity's. Office of PlanhiAg j Rest^arch Vf 
and Eval; nation is develqping a carefully'*structj^ed^>tonp '^fy^ 
ri^view procedtire f or the au^arding of con tracts, ;tb. Evaluate social 



I' 



progr^..' .■ : : , ^ _ /f^ ■ , ^ ■ }y 

.One unusual featur^^ ll lias recently employed ia^ the division of [ 

9^ ^ / 'fe ^c-' ■' ■ il' 

evaluation into tWo phases • \ During phase I, /the pi^limihary ^^V 

design that had beeu pr (Spared in*^house is;,<!biq>letea by outride \ \ 

contractors^) Phase II is the actual exj^ut ion of the eiiraluation* Tha i, 

I and JLI togdt||)ery^alth6^ 

proposal only for phase I. Three\^brgaQizations are awarded phase I / / 

contracts ; the one that produ^i^r the best plan at the: ^nd of phase i 

is awarded tlfe phise7lj:;jc©n . Phaae-Ir-thera f or a V beco t nes a n _ 



extension of th^ .convetition t^at permits better cbnparisoii 6f the j 
proapective performers than a promos al alone <|oo8; yet compcns^lies tlhe 
, contractors for the- added effort required of tliem. - ^ ♦ 



Anibng the problem arising from t^e use^of strict^. c^iq>e|;iti^' 
prdcorenent Bystem Bxei ^ , • ; / 



contracting personnel apply strong preajsiure t6 select; / 
«^t^ie iWest c6iLt biddety^ e^^ experience Indi'citjj^ 

ga.i^ogeth^ti 



■-r 



that low price and 'poor perionoance 
B]PPs inust b% widely distributees which leads^ to^ a 
large nutnber of responses , even for sipalL c^ 
The result a waste of contractor jpjs^d^ (la 
8um» scmetim^ greafter tbaii th^ CQutra^t's vALofe)^^ V 



JBnd agency staff tttte; / 

^ , • ..■ -r- ;• 4 ■ ^ • r ■ .-.;;>^v 

the ways that, the nuiaber of low qu bidders ca^ be 

x^>duce^ ±B tj6;;eiq>loy a. polnt^syptem in proposal^ «^ pwts^ 

a heavy weighttng/o^ ^v^^ staff quality 1 

■ ■ • -—-lev « 




pTOjii?^ acdbrdlrig^ 



^grpj.ect Motiltoriog . 
The task^f rrp 

to whiathex the proj€^ct Is l^j^iSilj^S^^ or it cduttractl ^ 

Grants aj;e asjsuoi^d tq.ittpps|& fe^^ 

and» therefbre , to ^ciq|>lire jless] j^iimrslsfht than a* contract* j 

^Grants « The usual>f»^^tern of ^ grants' awarded jni 

support of bade research /activities (inany of vhich are 
' 3 to 5 year periods) jls f qir eac^ to be 'assigned to a pr^^ 
(who nay have responslbiiiLty for 15^ grail ts) who reviews the gx aijit 
annually on the basiis of la written, prbgress report. He checks^he 
reports for substantiye jprogr^^^ for unsfheduied expendit^^ 
and for changes in the'cksti^bution of expenditures from. budge t:ed 
line.iteiDSt (Expenditures are presented in terns of accountlnjt 
^tegpriee ~ saiari^a^' e^ulpl^^^ in terms of jproiect r * 

actlHtles datJt gathering, expexjm^tal design, data analysis.) 
He fftay ^ alpprove snail (iioX) supplemental grants , but larger tones will 
bie sen^t back to: the review panel. No explicit effort will be oiade 
to~"kffep trade ^^lE ^piS&l^^ 



inventions mustl be 
^Tto jprovide a final 



^reported sep;arately. Eadi grantee is expect ed 
report at the end of his-granf-and ta stibnlt 



renewal application if he wishes to receive further ft nding; many 
*grani9«ed do nbt submit final report^^ howeyert . * 

The NIMH Mental Health Setvlc^si R&D Br^ch has str^gthened its 
grant monitoi^np prodedure^ significantly ylth the inte/it of increas-* 
ing th^ nu89>er of grants that produce results that are disseminated 
and introduced into |>r act ice,- ' i*' 



ti Frogram officers at i;^ visit grantees every 8 ov 

' ■ ■. ■■ --i ■ / ■-■ - |- " ■■■■ C- y-^' ■ 
9 months*' . / . ■ -i^: ■ t 
■■ ' ' ' y " ' V j ■ ■ ■ ■ 

o * E^ch |:^ranch qhief nsaint^ins a; manil^Hng^caH^ * 

in his (if flee on which 4fich project iSiliusted. 

^The data inipludest project, state^ Investl^atpr, 

progi;am niu]a|>er, -date^ rcseardi progress". The ^ 

, .pahQ^ review siM&ry (^^plnk* sheet") lH also 

'...attached, ' ■ ■ ' v. 

o Progress reports are reqjuired and monitored. 



But: the most Important ^^^ w^ inndvatinir is the fact that- 

^plannlng^ for and encouragiing diss Bmlnation - £u:id ii 
are eiq>lfa9i2ed from ^'the yeiTT begiimf^^^ - 



o * With his api^ricatlon foriti« each grant applicant 



receives a questiohnaire 



dont^alnlng the qiiestlons 



that will he i^xpected to ! be answered in the final ' 
report of" his grant (if receives one). These ' 
jquestlon.s put special emphasis on ihformatipn 
llilisseniinatlony use of prfi»4^<^ results, and 
; ||potential' users. At thi^ point, the qiiestibnnaire 
Ms solely for the ap^>llcimt^s Information and. vis^ 

il^n.^pii^ihg^ ; : [ 

^$lx months after the j|rane :1s awarded, the gt^nttse 



lil^eceiyes the sam^ qtie^tipnnalre with the^^ri^hder 
l||hat the final zeport jshpuld cover, the questions. 
j"i|lt cphtaip^. ■ ■ ' • - ■ 

At thfe end of the project ^ the same jqu^^stionriaire 



cSompleted and submitted < 



i^: sent; this i;|lme ^inal teport 'mMst be 



the report is ^xibmlttody it is ^y^luat^d by 

v.. *♦ / ~ 

executive secretipiry, and project officer 



J^) ^clarity of 



results 




'\- 



ThfK^resul 
submittii 



(2) ' cogency 

(3) worldlwld'e significance 

(4) ondunt^pf dissesdnation and utilisation. o 
of this c^ser lapnitoring is that the percent of projects 



•Vr- 



:yERic 



final reports containing usable infonoatipn inct^ased from\ 
AOZ seyei^al ye^rr ago td S^Z now. The saiDe percentage of finaL \ 
ow giy^ indicaiions of research utilizjat ion. The questionnaire 



reports 

is f^t ^o b^ a definite success in encouraging liiore dissemination 

actlVitli^s/ln projecti^« -v' ^ ~ 7 ' 

fcotjti racts . Tber^iJLs a greater diversity of practice jeji5>loyed in 



noni^^g contract progress than in loonitoring graitt, progress • Soioe 
ptocedures are quite extensive and detailed; for exan^le^.^^^^ 
by' the Air Force System Ck>iniDand in.inonitoring weapons r systems develop-* 
neht* Others approach tW Ijiforiik^ 

research jprc^ 

biannual site visits by the projebt offip^^^ interim progress f . 
report^ ai^d a final report at the end Qf tWelve months, it a contract ^ 
is not >rovlng to b^ ^rodueiive» it may -be cut;off . (In contrast 

. to a grintV for which termination 'would b.^ ve^ ^fficult;) . » 

Id the. case of the National Cancer Institute's programs being ^ 
manaj^ed accbrdlng to a convergence plan^ . contract jmonitoring is evcui 
closdr.j Zach ^'segment** of the plan is managed by a segment ^Halr- 

rman;vhe|m£^ be running several different related contracts . • These 
are monitored by a project o*fficer; one of fleer, pdr contract for 
large tontracts (say $2 million) ^ one of f icer pelr 4i or 5 contracts^ 
for smaMer contraets (say $100 »000 or $200,000) • All contracts are 

/ visi ted^ at leaiGrtr once'per year, soiiSe~^rBnceive~-aarrmany7 
' -More con l(:rol is exerted in the e^rly stages of a con tract ^tee 

4 times a* 

:v /officer si 

•. officers in the segment and the segment chairman. The segnv 
GhairiD^ .m^^^ to the KCI Associate Directq^>fh charge of 

thek prograi 



the contractor submits a progress report* The project 
risies these reports and -'circulates them to other project 
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J Each conttactor submits a new proposal* each year, ^hich is ; ' 

\ j : subject to the i^:^^ review process as ^e first year contract fac^d* • 
^ ■ ; The contract funding 'level nay be adjusted according to prevlofis, 

results. Horeover, the contractor's pterformance is recoipded faj: ' 
" ref emice in rahkljig .proposals • tt' has been found to be essential 

. lo evaluate 1>otlf the Individual investigator and the^£inD; the fonaer 

ieening to be the ^st lioportant^ . ; 4? « 

In addition to individual contract (nonitoring^ the converkf^nce 
process is distinguished by the attention it devotes *t;o inoni4:o:tring the 
progress of ^et^ entire progr^ via the convergence chart. 

The chairmen oi all the segments in a convergence plan^ 
t^orking group (the dore of the program management grou^ 
it , per month to discuss; pcogxesa reports « decide if the 'plan should 

reviewed, dlac^ programHEtindsv ^nd --related * 

^ BvbjepXB* , Progress is Indicated on the cdhyergence chart. 

^ <^ Once a year the working groups assess overfall progress and 

reassign pric^ities among flows. These 'priorities are anhounce|t ^ 
' \ ^ . and new pi^pesals flow- b^c^^^^^ 
ji\ ^ 1^ / * the proposal; as rioted earf^ - V-,-^. / 

contracts are also clbsely monitored. TWo or thrive H3E0 staff 
may be r^uii^ Ifor a million dollar ; two or three year eyiluatipii.' ^ - 

OEO's basic to have the ptoject' of fi^e^s 

p^jl^icipate in and'V^ers'ae id substantive w^rk done eri/the proJ|^^^ 
^e Bvaiuat^l^pn DiyisVm will not let the Contractor $ione decide ' 
wh^t will be done <m^i^ eyalU^ contractor^ ^taf^ ^^^ OEO project 
"^^officers collaborate oA developing Questionnaires and analysis plans^*: 
^^ ,1% ^^^ design, phase; Subsequently » OEO monitors -both the contr^actor 
^ ^yknd the i^ltes of the activity being ^^e^ 

A"^ogre«sea-^.'M rules may be modified. _ - / t • 

■ • Gene^L p^estions i ■ ■■-^- ---.-^ ^ ...^^ 




, ^ "U^lt^^ almost all of the . 

.f»*dlfierelK^^^ procedufes that have been described 



* " thus £ ar Ind-i^^^^^^^^^ can be discussed in separation from them. 



One is the\9ue8tlon ot hM^fhleation panels akouLd be oonBtituted^ 
The other is what criteria ehauld they eriiploy In making selection 

decisions. ' . . . " ^ ' . . , 

■ ■ ■ V ■ * ^ * " 

Panel ComposiClbn ^ The exact conqposltlon of a review panel / 

* • ■ . . ■ ♦ • . ^ ■ .. \ . 

vill^ of course, depend on tHe subject area of Its concern aiid on ^ 
whether it Is en ad hoc or continuing panel. Certain charactezlstics 
of such panels^ have 'general relevance t however*^ 

The flr^t' panel chosen in a new' program area ftas a dlspropoftipn** 
ate effect on the eventual quality, of the program. (This is obviously / 
/even more true of the cumulative influence of all the first pismels 
chosc^ on the eventual quality of a new agency.) The reason is^ 
th&t 'the panel mei^ers constitute a network Within the "invisible 
college" Within each discipline.. They are the contact pQints for 
many who have heard of the program* (or agetfcy); if they are not goba»{ 
they wiil not be able ^ to attract the TbeTt of ^e^^^ 
progrfon. Ar^ianel menbiers rotaf^g^the network grows and expands 
knowledge of the^programj. FlveTyekra may b4 required ^or a new baslt 
research program to "tail^c of f" W this way. 

KIH study si^ction wojpkloa^j are cbntinually ^nitored* ^ If .its 
grant applications dwindle » a stWy section may be merged with ^ 
another one; ff they grow it Bay In any events 

each study section's authority expires after 10 years { it must be . 
rev^^^w^ to decide upon con tinu^^^ ' 



At . boQi^IH and NSP^ it is felt to bte essentl^ for. the staff to 
pitk new panel members. At<kNIH» the executive scicre€ary' and study . 
' Section chairman recbimDend 4replacements^[ who ^ TO be approved by . 
the NIH Director's of fice and the Secretaty of HEIjf, At NSF^ the ^ 
pro^ram:^f fleers and section heaps, ^^^commend panel members r who must 
be approved up' through the A^ Director's Office. ; V 7^ 

' r.^^^ mixtures points to a nUkiber 

:.of pbt^^tial; problems ; v ^^v; 'i? r ^ ■ 

o Panets/l^ repx^duce "the kind- of _r0searoh their . ^ 
^^^^ m comprises blol- . 

ogi8t*s jand blochem^i^ pyoven hosplti^ble to 

^ " blologii:ikl- while rejecting, 

for ex4Bple, comfiwnlty research. ^' ^ »^ 




Tig norwaientiate panels has hoth henefite 
oastB^ If they participate actively,' t[iey 
provide strong prejB^ure for relevant work. However, 
many are dominate by the sclentlata on che panel ^ 
£n^-8ome scler^^ista^^^^^ resentful, that the nqn-* 
/Scientists cannot share in the extra vork^ of 
primary reviews (careful reading of proposals before 
the panel meeting), 
Jntez'iEaoipHnaxy pai^ are heir to cC ae^ida of / 
itts reeulHng from problem ^of prestige am 
oormmioati di^oiplincfS^ For exapple^ 

on KIMH panels- soi:ial p8y(^oipgiats and anthro- 
pologists find commun;t^t:ion difficult with 
cUnical psychologists and biologists » not^o say 
pfiysiMsfcs' and edi^ 

scieipLbdrsts are V^ry .jBtrx^ (lespiecialiy with regard 
to methodoiogy)' cm/they 

t6^ bipiogli^al ^cii^ntistsv jwhile ihterdiscipllnkryr 
paiiel^ .might be th l^e Mre open^^ t 

propbsieaif 1^ a^^ discljllihi^, this is ndt like^^^^ 

to be th^ ciase If i^ th^ disiirtpllhaxy r^^ 

^ the pan^ is strong, for Judgment will dominate" 
^d tenet to b^ accepted by ^is cblleagueev: In a 

^ panel of his ^een> however^ might: be* 

dmHej^jg^ . -Thus , for singlje d^ propqsals 

/interdi^i^^ p^els ma^y* be less qpen than 



.Proj<^t Selection triteija , ' The prlMipal crl^^^^ by 
panels in mirfcing ^r^t ^^j^^ ' ; » % iv 



if !- fr^^ compe tence of. prinaipat inveotigator. | /| 

**Witfadut doubt the st^rbngest Inf luence on the pan^l ' 
decision is the reputation of thc^ principal 

V .^"^^ investigator. 



I- 



*o ] j^vimB ezpein^enee in ^ 

• If the investigator has no^^^mox jpiJbflLcatlons in the 

• area, the proposal iSx^Xikely to receive a bad rating, 
o Adaqmay \of fad It^Hee txnd other, re^fiureee,. 
p Stretwth ofj^Toaah and Te^iicxtck * : 

i^^epetitlve? / Does It add tc^W:i^ntiflc , 
^ .owledge?^ y,^ - , { ■ ;. • " • .\ \ # 

o / EBlex^axioe to ppotlem area; , ' • 

> ' ^^-v ^ ^ -o' ... ' . • ■ 

Potential for me ^ - ^ : / • 
/ these last V two criteria 'are e^t^aslted in applied , 



teaeaie^^ bu.t not elsewhere.} 



o le budget n^onableZ^ 

This is gener^ally only an ^ntfecedent «to negotiating 
the budget dovnvard if nejLeu&axy. . * 

Ihesu criteria are reasonable tJij^s/ ye^ 

econoioiist *V point-bf-vljewi^w ychpy aiife concern^ almost exclusively 
with thg likely benefits of thy( jSzesearch and not with the relative 



cost of achievl!^g thMe bieniefira* (Only when «oae^^^o 
scoring nodela^ are enployed £a project chb^ soi&e balance' of cost" 
against benefit made.) In tte eta when resources were growing ^to^ 
rapidly^ thail quality rese;i^ii6> # swch selection-procedures nay have" bew 
adequ^lte; But- when tesout^s ai^f^ s^^^^ compared with Aj^at might be 
dbne , both, costa and benj^ltfe 8h|vid affect the dcrclsioti process • 

HOW CAN tmCVAH^ALiJAW ? . 

1^ the achievewBntp 

ofthe R&&oP7ogtdm snoul^d b^ as cert lined and compared with its 
itiltial goals. ^Th&;^tes]blts of the^^e^ affect the 

revision of 'the pr6gram^ or the design of new pTOgrams . 



Program evaluatictti ^^^^^^^ seiise is ^difficult to identify as a 



s'tfparate phase of R&D md^^agen^^ It shades imperceptibly Int^ 
project mini toflng-a^^^ 

d^elpp^t' phaseAand Inta |^^^^ Our intent, howeVfer» 



' is to di^^tingulsh those actlVitle)» that examine the achlisyeiDentB :Df 

• thj^ progrto 00 a whole ^ and nbt siisply thdipe of its conscltufent 
projects, one at a time* Moreover^ we seek to separate those actlv- 
It jles that undertake 8 usns^ry judgron program achievement at a 
points in time from thpae that attexiqpt continuous moiiltoring of 
progre^a.. With these stringent distinctipnsy we ilnd that J^fiJEM^ 
oent agencies -currently devote very little explicit activity to/ the 
program evaluation phase of management. The extent ^nd nature of 
such activity ^differs according to the typd^^^^^^^ IJiStSr being suppo;etod. 

. ^ Basic kSD , ' M . 

. In the case of the'^ agencies sapportlnig basic research, the ' 
^ genev*ally held view Is that thd best judgments o^ pro^ire^s in a : ^ 
- 4>ra^am area cosee from the scientific comnnmity Itself and, rnqrebver, 
that these evaluations should be xaade' «^ regard to a^research 
stddject are^ as a ^ole^ not^ just those projects vithln^^ that 
hajppj^n to be supported 'ty the a Thus>^ when ex^^Hclt^^j^^ 

. evaluation occurs, it is gerte rally In the form of a 8t«Lte"-of--the-art ^. 
evaluation chat seeks to ^review l>ast accomplishments and present > 
knowledge and to identify {rromi ding Cons^uently, 
evaluation of ;>riol^ ptograw and planning of future pt^ograms 6x^^ : , 
Occur together* y - : * ^ ^ ^ 

... . . • . ' ' ■* -t . . » 

o- At NIH these state-of-the-art asseis^sments are often *' ' \ 

* \ carried out by workshops sponsored either by the 
* study section^ or the^v.i -1? 

V ' llle^irational Institute of Dent^^ : 

* f6iN!:je9!bmple, prgani^es asseass^nt workshops on • 
fair^.y fli^eclfic research topics? the 
• ^nctics- of^clcftf /palate or dentaT amalgam. * 



'v . - 



Ordinarily, each^progridcm director at NIDR holds 
one or two such woiicshops each year, returning to 
each topic everry five or so years'. Attendance 
compi^ses 15 panelists selected by the program 

director 'to cover: ; - » . 
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b" 3lll^;slgnlfl^rafe^t reaearch approaches,^ 

- o . practltiojrers associate^ with p 

> ^ aspecta c^^^h^ area,** , ^ 

o foreign, and indimjA^ reaear^ 

' -i jiave cohtrlbuted,> ^ , 
... \ . - ' . ' 

\; o new lines o£ research, and ^ 

- . - A -©---elder ataft^aien. l '^-^ ■ 

.Porraal' state-of-the-art papers arc proscnfcd -ar 

^ ^^t&wWkahop, followed by ^discussions y and ar 

; - . att^t to su8imarl26 promising dit^ctions of 

i^utuire research-; ' ' ^ * . - ' j 

: y ; < ' - ^ 

Jud^^ntsV^ncerain^^^ success of tan ^Igency's ovn prograss 
(not: just thfe auccess of th^ research ^rea they«are supporting) seem 
to be primarily\ implicit and associated with ^ersonn&t^ and budget 
decisions. . . ^ ' ^ 

Each l?ranch\ chief, sfitction chiefs dlvieidn director, and^ ^ 
i. institute head shpul d' liiake such judgmen'ts in Evaluating thct quality 
i p£ the e£f<^rt8 of his subordlnateB* Uia principal neax^ of influence' 
land 'fcorrectioif then is through his selection 'iind' rewarding of his ^ ^ 
istaff menbers* But at NZK» f6r example/ the shape of the' e^^^ramural 
Wsearch progrdn is really in the hafids of the study sections gF^d 
tp a iessar 4egree, the advisory councils; staf f meBibers havp , 
j^aeively little Influence « At ONR, however^ program responsibility 
does re^ide^ cl^^rly* with the program "bf fleer and evaluation of thet 
p^gram generally occurs lioplicitly as part o£ fivaluating the - . ^ 
p]^ officer's performance* Thus, while prc^Sram evaluation is' *^ 
implicit in personnel^ evaluation. Its relevance varies considerably 
anton'g agencies. ^ 

' Program evaltiatlon also.must enter' lmp,llcltly into budget 
decisidns , not only vithln the agency . but in its dealings with Ita 
. superior agencies (IBEW ot DoD) witlt the Of f ice oi Management and 
Mudget «(OMB) , and with the Congreas^ With respect to basic re*search^ 
these judgments are ordln^rtily very sub/jectlve and cru^e. At be^ti 
the numi^er 'of scleatific papers publlshod as a result of a program 



» . » 



, • ' ' :^01~ • , - : ■ • C 

- V ■ «* ■ 4 ^ 
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, or SD»e listing of aaajftr f it^dings -vill be displayed a ineasure, o^. 

The difficulty if evaluating ba&ic rese&rch programs is consldfer-^ 
able and underlies the absence of formal methoite or* procedures ^ This 
^ i^ especially ^rue of eval^^u^tion with ^elaipect to the criterii>a that 
0KB or ^the Congress ml ghti consider appropriate: ccnitTibution Ut the 
public welfare. The, time lags and unp!redlc table relationships 
between new basic knowledge and Ijiqproved technologies, or public 
services oiike such eyalxxation virtually impossible* , One step resx)ved 
is a criterion that would be somewhat easier to employ £^d that ^ 
scientists 6nd i?&D mnagers might consider appropriate: cx^ntribution 
^ knoDtiadge in the field . Nevertheless^ it itometlt^s takes a ^ 

* considerable period before even the scientific value of new'-dlscov 

eries becojfoes plain^ A third crit^lon fbr program evalyatiui may^ 
sometimes have to substitute for the preening tyor ult concerns, not 
the consecHienc&s of ^ 'program » but the process by which it was 
constructed: adeqmay^ofB£Dm<magefrientprQc^ 

of this -case would-be that if first clasci scientists are c«^iil.ted» 
. if proper selection procedures are used^ if adequate in^orttiitioii is 

* availa|^le» and good staff meters are 'eoqployed^ the resultant. program 
.is likely to be %oqjt:- Vhile each of these criteria enters implicitly 

into, the personnel and budget decisions made cbnceming basic 

• * research I to our knowlecige none i« employed in explicit / regular y 

evcUuations conducted as part of R&D management,. 

■ ■ • •< ■ „. •: ■ : 

PractieaA^&D . 

> In* the case of agenoles %MPporting practical R&D » it >is 
appropriate to evaluat)^ i^ogramar thrdugh thefF' contribution- Jto 
improvements in practice, . Novertheiess^ we have found oniy a few 
instances in nhich R&D agencies identify the evaluation of thejfx 
' * . programs as an explicit: part of the R&D management procesfr. v However* 
since these progrtnasp tend to be continuous and linked to the practice 
community* some evaluation probably occurs oo a cojixinuiing basis^ . 
throu^ feedbadc Ihrom practitionerE and consequent adjustments 
in program plans, in the •atjri culture ca^f^, ^fot CHfiRiplc, there .are ' 



numerous links betveen fanners and other users and the -R&D system 
that' serve to let the R£J) program managers know how well th^y |qre 
doing. In addition, the same kinfl of itopiicit program evaJLuatlLon 
during personnel and budget decisionmaking that occurs with respect 
to basic research -do jbtless occurs in thesis a^eas. 

Perhaps ttre xoost expllcU and regular procedure for program 
eval^uatio^i that, wp discovered is the one employed by the Mental Health 
Services R&D' Branch of NIMH* It is eotf.eDdluqtion in terms of 
^clf -specif led goals. It employs a formal "Goal Attainment Scale,". 

which is a grid in which a program's goals for the year (usually. 

* ■ 

five or so) a^e the columns and five possible levels of goal 
attali\ment are th<i rows* The rows are assigned integer scores from 
-2 to -K2t and th6 columns a weighting of 0 to 10 accor(Iing to 
impoiNb^^*' Within the grid cells £|rc statements of what would 
have to be achieved for that level of attainment of that goal to 
be reached. ^ 

Each year each progrfl^ director (thejre are 16} « irf consultation 
with the branch chMJ^^dF^th* branch staffs decides on his progtam*s 
goals for the yf^ar* *The program directors and branch leadership 
revise the grid quarterly* although progress is evaluated by the 
branch chief only orfce a yeai:; Moreover, progress^ls discuissed in 
Joint sessions of all the program directors twice weekly. Joint 
sessions reduce, the likelihood of self delusion. Attempts to record 
progress narratively failed; the grid method is preferred. 

The character of Agriculture *s extyramurajL research, wlhlch is 
almost exclusively carried out by the 53 State Agricultural Extension 
Stations under -formula grants from the Coo^pcirative State Kesearch V 
Service (CSRS) and other sourdes , makes formal program evaluation 
difficult. Instead, what has de^;;i^opGd a system of insibituiional 

program^ evaluations f in which CSRS reviews the activities and plans 

J . 

of each experiment station^ at least once ev^gry four years* Reviews 
are conducted as two or thee day site visits by a CSRS program 
director, a CSRS visiting scientist, and possibly scientists from 
other stations. The nature of the evs^^uatioiK has evolved o^er the 
jyjars, but it ^now tends to be con cerrted with all the stati<^s's 
progVams and to focus on future direction for the programs, although < 



psjst^ actlvlt;ies sre nedessarlly reviewed^ / 

/ EvaLuacions'may also be conducted by it/dependent outside 
evaiuators. One suoh "activity: Is a tiurreric review of 8-^10 areas 
pf agricultural research b^ing carried out' by the Natinal Academy * 
/of Sciences. The emphasis in the reviws Will b^'on the quality of 
science in those areas; soioe attention will also be paid '^to the 
quality of the research environment and research personnel. The 
reviews will ^e conducted by- separate panels of about 10 mesbers 
eadh. 'The panels will examine each, pi'oject in its field and evaluate 

the adequacy of funding in the area, tha appropriateness and quality 

' * ■ . ■ - ^'-.-^j' • - • ■ ' 

of current; res^^bt on 9 'and future directions. 

.■ - . ■■■ • - " i 

Programmatic R&D ^ a ♦ ■ . ^ 

In the .case of agencies supporflng programmatic R&D 9 a program 

evaluation is'gc]:)rerally an obvious and resjsential part of R&D\manage«- 

ment; for» by' definition, programmatic R&D is concerned with the ^ 

achievement of a concr objective ii^ a finite time period. Rough 

" * * ^ . ■ ^ * ■' • •■ 

, evaluation 6f a program is immediate: does; the spacecraft work as * 

intended? is the weapon system operational? has a vaccine for 

• ■ ^ ■ ' " ■ " ^ ■ ■■■■■■ :• . 

leukemia been produced? 6±d the new pfroduct reach the market as 
plahticd? 4iare refined evaluation is generally more difficult , for 
it. should consider whether iShe product pei^forms in* all^respecta as 
desired and whether the R&D* program took longer; Qr cost more thW it 
might ha'^e. These latter questions rarely seeM to be examined ^ 
explicitly by R&D agencies themselves*^ If raised, it ^een^ to be 
most often as part of an investigation by the OMB.^' Congress, , 
.the General Accounting Office triggered by the poor performance or 
cost overruns of the R&D product. 

HOW cmA .Institutional development be managed? 

In addition to the basic mode of research project support that 
has been discussed thus far^ a number of R&D agi^cies have found it 
desirableL, CO create and provide broad ^s»inscitutional support to 
nmi extramural R&D indtl|^tiQns > generally called J?dU7 eentera. We 
shall examine four major issuesr associated wixh the establishment 
arid management of such centers: ' ^ 



o How are they organv&edt J ^ 1 • * 

o What purpoaes do.they ^serye?^ ^< • ^ ^ 

o What factovB omh^ute to. their BucQeasl • ^ 

o How zaji th^lx T^gresa be nonitoi^ ^ 

Organisation .1 ^ 

.Most R&D centos are organized at universities j many In ord^r 
to foster Interdisciplinary res^rch on a topic of concexti to the v 
sponsoring ;agency. ]Por/exaxiq;>le » the National Institute of Dental , ^ 
Research (NIDR) has supported the' establlshsaetit of a nwbftr of Distal 
Research Institutes aod^ Centers at universities, but outside of dental 
schools I In J>rder to broaden *the/range of disciplines engaged in the 
atudy of dental prbbleins* SlmQarXy, ^he NIMH Institute of;^^ Alcohol- 
ism and Alcohol Abuse create^ a nunber'of 'university reseair^ 
cetiters on alcoholism* Us part of its effo](4:/to:bxiild the conmunlty , ' 
of Jresearchers" concerned with alcoholism* ' " / 

^hese centers *f requently are built around a central figure, 
usually the dli;ector, who Is given multlyear institulllonal support;,' 
and the Charter of developing a viable research progrmj^ x 
in' th^ resiearch area*" ; ■ "'-/.V • : * • 

The NIDR Dental Research Centers are esqiected to haye b(^h an 
inetitute policy comAttee^ to faeilitat'e w^;^ guide thct 
of* the center in accordance with NIH and iinlversity poilcie ^ 
a eoientifia peuietJ oomrAttee , to oversee the fuali ty axul cohesion 
of the scientific program* , ' 



Purposes ..^ 

Extjramural R&D center? are created to serve a number of ^pi^cific 
, purposes' Ih^udlng; : . ;,.„ -.^ 

o to help" in recruiting highly c;p^ ^ 

problem drea by pr6vl4ing stsibXe support ao^ giving • 

it;-; visibility,' ' ^ ■ ' ■ .T 

o to demonstrate the urgency and priority of a problem^ 



f 
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to focus a variety of taLents i&od disciplines) » 
who would ordinarily wbtfc separately, on a specific 
problem. 



' 0 

Success Factors 



Among Jthe factors that are felt to Contribute tp^he success^^of 
R&D center are: • 



o The quality of the dLrefOtgr^ The NIDR Insists on 
. 'knowing who the dirictor will be before approvlhg 
a new center^, alrhough the real problenii Ts^^toTt^^^ 
^le to anticipate the director's perfbrinlbiite 
V* * before he Is Installed^ The illDRf ajbira^qi^entlyt 
^ would like to establish training j^ograms in eat^h 

center to find potential directors and test them' 

■ • " ' / --^ ■ ■ >■-•■ ■ i ■ 

' In subordinate positions. The pro^lSm in ^ntal 



i 



'-.s^, "^Z research, as It is In many otl;idrrrf l4g^ :s 

. " ' there are few scientist directors ». . 

iff intierdlsctplinary xesea^ch^^^c^ ;^ 
^5) ' The quality of i^e BpientCfia reideD oonmitiee. . ' ' 
* . At the Dental Research ^ntera, these commltteea ar^ 



\ 



- concerned with dn-gQlng prbgram review, new proposals, .^rV 



center's p^bleoi area« ^ 



^ogress Monitoring . 



n 



Perhaps the most, crucial factor in th^ success of extramural 
centers is the* means by which their progress Is iiiohltored and reviewed. 



and the hiring of new '^researchers. . • 

o j^ The univemity^a oonmi f 

c^ntaK This commitment ;^ould h.e not only for ! V'ik 

■ " *■ ' ' * ■■■ '» , . • ■ ■ ■ ') " ^ ""' ii 

resource^ administrative attention/ but for v f . y 

'joint appol^^ ln\ university departing -f 

" the center's staff; Joint appointments have; 
V. proven tc be necejssai^ to recruit gb^ ■ .i 

I ataff ; thejr have the\addlti^^ ' ' ■ W*' '^ '''' 'T 

> interesting other departmental \8clentl8t8^1n che 
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The NIDR has develdped apparently successful procedures for Bonltoring 
and reviewing its canters* . * ;( 1 

The centers ar^ given assurance of support for 411 initial ddvelop-^ 

rmnt period of five 'years* The asiouht of r upport is negotiated elach 

\ " . . ^ ■": ■ . i' - • ■• ^' . ' \ ; 

year in accordance progress achieved and funds, ^vailablei^ \ 

Since tlie centers a separate lii^e item in'NlDR's;iu(lget^t^ey dp 

not compete with othe^ dental research for support; they do cosq)yet^. 

with each other^ however; 

* Every center is ^amined oamually hy^ a NIDRrappointed pane^- 

prominent scientists dental- related disciplines and scientist- 1 

adiainislurators. The panel,'consisting 61 14 or 15 menders ^ spends* 

2 days at the center with! the pzrlndpal bbjective of reviewing thein 

A • ■ : - \- Y\ - • * , • . / / • \\ 

scientific merit and^lJie direction of deyalopotent of the.center's ! \ 

\ * t ■■ ■ ■ ' \ iV ■ '■■^ ' • ■■ / -"^ T' 
.program. Ihey wila-also se^k tb find oat^ |how well knq^ and/regat# 

. the center's actlvitlesj^e «n ^ampus'^Vad hol^ well it 1^ admnist^red. 

After an evening executive session^ the panel will spend a day 'aiia^^ 

a half condiicting tn^ xjevlew/ flrs^t in^a larg^ laeeting and /then iii 



smaller group and individual discussions. Several times the panel 
will regroup to be sutej that all topics ^re, being covered/ > Plt^^Y^ 
tile final half day , ah loutline the review, ireport is ^^ o^ 
the panel members subrnjlt reports on^ their specialities 
At the end of the visitji ^e ttnlvex^ty is asked li^ It is satisfied 
with ^e review (and, j^tivatelyv witli the tev£ewktB) • 

The NtDR*s Dental! Re^search Institutes and Sj^^cl^^ Programs 
Advisory Committee me^ia cptrimlly to review Ttijey. 
are presented id.th. thcf panel reports , previous repo^S> and^^^^t^^ 
centers' Annual reports, often in summary fora Th^y spend sejveral 
days reviewing all the craters and meeting with all the centeri 
directors* Their judgment weighs iieavily in the/annual budget ' 
negotiatfpn with the ! centers and, espe^ylally , Itif the decision jas 



to whether to provide support beyond .the 



4 • 



feERJC. 



V 



/ 




